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ABSTRACT 


This thesis reviewed 45 studies that purported to 
empirically assess the effectiveness of information systems. 
They were evaluated for (a) vulnerability to alternative 
explanations of reported findings, (b) effect of data 
collection methods upon their conclusions, (c) capacity to 
establish causal inference, (d) how they defined system 
success and (e) reported changes in organizations following 
information system implementation. 

MIS research methods do not provide a basis’ for 
establishing causal inference. This, in turn, denies the 
possibility of constructing a formal theory to describe, 
explain or predict the successful implementation of 
information systems. This deficiency is worsened by the 
emergence of antagonistic schools of thought among MIS 
researchers as to how system effectiveness should be 
evaluated. 

Recommendations are made for improvements in MIS research 
designs and data collection techniques. Suggestions are made 
for ways by which line managers and commanders as well as 
managers of MIS activities must become informed consumers of 


MIS academic research on system effectiveness. 
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I. INTRODUCTION 


The point of implementing computer-based information 
systems is to gain either increased operational efficiency or 
enhanced managerial effectiveness. These gains, however, are 
more often asserted or assumed than they are demonstrated or 
tested (Lotus, 1988; Fawcette, 1988; Bulkeley, 1987). 

Starting in 1985, senior line managers and information 
system managers in the United States began to ask whether 
they were getting a payoff from their investment in 
information systems, especially since the advent of the 
personal computer. 

A similar skepticism could well be evident in other 
countries (Maginnis, 1986;Bowen, 1986; Strassmann, 1985; 


Wessel, 1988; LaPlante, 1988). 


A. GROWING DOUBT ABOUT THE PAYOFF FROM INFORMATION SYSTEMS 
Steve Crummey (1988), of Lotus Development Corporation, 
assessed the current information system productivity dilemma, 
at least as it applied to the proliferation of desk top 
microcomputers: "In 1976, at the dawn of the personal 
computer age, corporations invested $100 million in PC 
(personal computer) technology. Ten years later in 1986, they 
spent $46 billion on PCs--and nobody really knows what those 


dollars are buying." 


1. Questions That Arise 
Seeds of doubt about the assumption that the 
automation of management information systems (MIS) leads to 
increased productivity provoke questions about the ways in 
which the success of these systems are evaluated: 


a. What is meant by a payoff, success or benefit from the 
implementation of an information system? 


b. What are the magnitudes of improvement in organizational 
functioning reported by formal studies of information 
system implementation? 


c. What research tools are employed to gather data on the 
success or failure of information systems? 


d. To what extent do the findings reported by formal studies 
of system effectiveness have scientific credibility? 
2. Answering Those Questions 
To address these questions, this thesis reviewed 45 
formal studies, ostensibly dealing with the measurement of 
the effectiveness of information systems. They were examined 
£Ox> 


a. How they defined effectiveness, benefit or success from 
a system implementation. 


b. How their research designs dealt with threats to 
EW el jess 


c. The means by which they gathered data on system 
effectiveness. 


d. Reported levels of improvement in organizational 
functioning following the implementation of a system. 


The findings of this review are analyzed for their 
implicataons fom: 


- Senior line managers and commanders as well as managers 
of information system function. 


- The intellectual foundations of the study of management 
information systems. 


B. THE NEED TO STUDY PAYOFFS FROM INFORMATION SYSTEMS 

After spending billions of dollars on automating the 
offices of white collar workers, organizations find that they 
must now scrutinize the claims by computer vendors about 
increased productivity. To effectively manage a resource an 
organization must have a means to measure its benefits as well 
as its costs. 

An appropriate managerial question that arises is whether 
an organization is getting a return on its investment in a 
resource. This necessity is not diminished because the 
resource in question is a management information system. 
Managers of information systems are pressed by senior line 
management to answer, again and again, the question, "What is 
the value of the company’s investment in information systems?" 
(Matlin, 1979). 

Pecording 1K: Strassmann CLOT G jy "Tndustries and 
governments all over the world are currently struggling to 
contain rising administrative and clerical overheads by 
automating information handling in the office." 

While the clerical work force has been the fastest growing 
segment of the labor force in nearly every industrialized 
country since the 1960s, the productivity of this sector has 
been flat since the early 1970s (Harris, 1987). This has been 
the result despite the fact that 40% of capital spending (in 
the U.S.) has been directed at information technology. 

General managers in the United States surveyed by the 
University of Minnesota Management information Systems 
Research Center (Brancheau and Wetherbe, 1987) ranked the 
measuring of system effectiveness as the fourth most critical 
problem they face in managing information systems. Sixty per 


cent of the Management Information Systems executives polled 


in the same survey placed the measuring of effectiveness in 
their Top Ten list of critical problems. The assessment of 
system success has been in the Top Ten list of critical MIS 


problems since it was first formulated in 1980. 


C. THE NEED TO DEVELOP A FORMAL THEORY OF SYSTEM 

EFFECTIVENESS 

Clearly, then, the evaluation of the success. of 
information systems is viewed by senior line managers, and by 
MIS managers, as being in the forefront of their concerns. 
While these executives represent the sectors who have the most 
at stake in the successful implementation and exploitation of 
information systems, in the social division of labor they are 
not the agents of measurement. 

That task falls to a neutral sector of the information 
systems realm: the academic scholars of MIS. Management must 
examine not only the effectiveness of information systems but 
also the effectiveness of the ways by which effectiveness 
itself is studied. It is the job of the MIS scholar wee 
discover and systematize what is known about’ system 
effectiveness. A theory of system implementation would express 
what is known, what is predicted and what is assumed about the 
success of information systems. Without a formal theory of 
system implementation, we cannot competently describe, explain 


Or predices: 


1. Functions of a Theory of System Effectiveness 
A formal theory of system effectiveness would be the 
basis for describing, explaining and predicting system 
effectiveness (Denzin, 1970): 


* Description--Formal theory permits the description of 
a phenomenon is such a way that others can repeat a 


study with a high degree of agreement. Formal theory 
can provide the descriptive data that can answer the 
managerial question “What is the payoff of information 
systems?" 


EPxplanataon-—--Formal Stheorye conse#sts» of th meet of 
interrelated propositions such that some can be deduced 
from others and, together, they make sense out of what 
is observed (Homans, 1964). To say that a phenomenon 
has been explained is to argue that its occurrence in 
the future can be predicted. Explanation assumes that 
the causality problem has been resolved: that 
covariance, time order and the exclusion of rival causal 
factors have been established. 


Prediction--Prediction is the test of explanation. If 


we claim to have explained why a given set of variables 
occurs together we must be able, by definition, to 
predict their future relationships. 


2. What a Formal Theory of System Implementation Is Not 


A formal theory of system implementation is something 


apart from what is usually described colloquially by the term 


bemeory . " 


a. 


A model is not a theory. Models are sets of relationships 
built upon assumptions. Some of these assumptions may be 
based upon rigorous empirical assessment or upon an 
empirical base of everyday experience. But a model itself 
is a metaphor; it is used as an aid to description. 
However, a model is not a foundation for deriving 
testable hypotheses from interrelated propositions. 


Conceptual frameworks are not theories. 


A single proposition, say, if X then Y is, by itself, 
not a formal theory. 


Nor do sets of propositions, by themselves, constitute 
theory. They must be arranged in a deductive scheme 
(Homans, 1964). 


Taxonomies are not theories although they may be a 
starting point in the development of propositions that 
can lead to formal theory. 


f. Criticism, journalistic observation or hunches are not 
theories. 


D. ELEMENTS OF A FORMAL THEORY OF SYSTEM IMPLEMENTATION 


1. Formal Theory as Interrelated Propositions 
A formal theory consists of a set of testable 
propositions that are related in such a way that some are 
deduced from others. That is, for example, Proposition Three 
may be deduced from Propositions One and Two while Proposition 


Four can be deduced from Propositions Two and Three. 


2. The Function of Propositions in Building Formal Theory 

The propositions that are the first level of analysis 
in theory construction are themselves statements of 
relationships between the next level of analysis, the concept. 
Concepts are the bricks in the building of formal theory held 
together in propositional statements. 

In the proposition If X then Y X and Y are the 
concepts. The use of concepts is critical in the development 
of a formal theory. A formal theory, like the proverbial 
chain, is only as strong as its weakest concept. Concepts 
represent pieces of reality. They have three functions: 

a. Introduce new points of view into a scientific process. 


b. Convert perceptions of an environment into scientific 
aCra vat y- 


c. Make possible deductive reasoning. 


3. Concepts as the Basis of Propositions 
Through concepts, everyday experience can be raised 
to the level of shared knowledge. Nonetheless, a specific 
concept is a commitment to a single viewpoint, excluding all 
others by its formulation. A concept is also a plan of action 


to observe a phenomenon. Concepts suggest operational 


activity and measurement. At the conceptual level, as shown 
in Figure 1, a propositional statement takes the form if A, 
then B. If user involvement is high, then a system 
implementation will be successful. 

This statement is put into a testable form at the 
operational level. It is expressed in the form that a high 
score on an attitudinal scale indicating user involvement 
leads to a high score on an attitudinal scale indicating user 
satisfaction. A researcher proposes a null hypothesis that no 
relationship exists between the scales. Specifically, that, 
Say, a Pearson correlation of the reported scores will be 
zero. 

The alternative hypothesis will be that some 
relationship exists between the scales. Again, the specific 
practice will be to reject the null hypothesis and accept the 
alternative if the probability, of a reported Pearson 
correlation (or similar gauge of the association of two 


variables) having occurred by chance alone, is less than .05. 


Conceptual level Concept A Concept B 


User Successful 





Involvement Implementation 


Operational level Indicator A indicator B 


met lL cucaina | Attitudinal 
Questionnaire Questionnaire 


Scale of Scale of 
Involvement Satisfaction 





Figure 1. Operationalizing Concepts 


a. Definition of Concepts 

Defining a concept is a critical link in the 
conceptualization of theory. A definition must specify what 
is included in the concept (not what is excluded). They should 
be expressed in scientific language, not in everyday, 
colloquial language. 

That means that the expression should have a high 
degree of consensus among its users as to what it refers. 
Moreover, a definition must not be a tautology. Rather, it 


must have an external reference. 


b. Nominal Definitions 

This is the primitive level of defining a concept. 
Generally, it means nothing more than substituting one word 
for another or employing pseudo-mathematical notation. An 
example would be the use of a capital letter S to be a symbol 
for the concept of self. 

Nominal definitions have no meaning beyond that 
arbitrarily given to them (Bierstadt, 1959). No claim to 
empirical verification is possible. A nominal definition 
Cannot be proven true or false because, by definition, it is 
just as it is defined (Denzin, 1970). Being true by 
definition, nominal definitions cannot provide a basis for 
causal inference. 

At root, they are useless for the construction of 
propositions for formal theory. Nonetheless, nominal 
definitions are not without utility. The use of pseudo- 
mathematical notation at least has the virtue of economizing 
on space in published matter. 

Further, a nominal substitution of one term for 


another that is burdened with emotional baggage allows a 


concept to be treated in a manner that is more scientific and 
less ideological. Pamnally, mominal definitions scan be a 
medium by which new terms and concepts are introduced into 


Serentafic usage. 


c. Real Definitions 

A real definition resolves a concept into its 
constituent parts (Denzin, i970). For ¢teample, arser 
involvement can be defined as the users of a system that 
participated in its development and implementation. This 
definition can be looked at in terms of users, system, 
weeecicipation, »dévelopment..and implementation. It points 
immediately to specific steps for observation. 

The attitudinal scale used in an example earlier 
would need to have sets of items that measured the degree to 
which a respondent was (1) a user, (2) used the system being 
evaluated, (3) participated in development and (4) 
Bercticipated in implementation. 

Moreover, a set of items would also be needed to 
eetaolish the weaning of athe term “participation™ to each 
Beepondent. The test of areal definition 1s made in terms 


of the empirical observations it can generate. 


d. Operational Definitions 
An operational definition prescribes what to do to 
observe a concept. It is the vehicle by which real definitions 
are translated into empirical tools. It is tantamount to an 


@eelrical indicator at the operational develsof a propesmtiton. 


E. CONDITIONS FOR ESTABLISHING CAUSAL INFERENCE 


1. The Function of Causal Inference in Building Formal 
Theory 

The foundation of formal theory is a set of testable 

propositions that, finally, can lead from explanations to 

predictions. That implies that concepts in a proposition 

must be expressed in causal order. The proposition, if 

Concept X, then Concept Y requires that Concept X be the cause 
of Concept Y. 


2. Three Conditions for Establishing Causality 
For Concept X to be established as the cause of 
Concept Y, three conditions must be met: 
a. Covariance: both X and Y must vary. 
b. Time order: X must vary first followed by variance in Y. 
c. Rival hypotheses that can explain any observed variance 


in Y must be excluded. 


a. Time Order 

Establishing time order between two variables 
requires that the design of an empirical investigation must 
account for time. Evidence must be gathered at more than a 
Single point in time. This suggests a longitudinal or time- 
series design. If the effect of an intervening or independent 
variable X upon a dependent variable Y is of interest, then 
the design must involve collection of data before the 
appearance of X (observation one: O1) and again following the 
intervention (observation two: 02). A pretest observation and 
a posttest observation around the occurrence of the 


intervening variable X is expressed in this notation: 


Ol X O02 
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A difference between O1 and O02, ceteris paribus may be 


attributed to the intervening effect of X. 


b. Rival Explanations for Observed Changes 

Things, of course, are not otherwise equal. 
Alternative explanations arise to account for differences 
between O1 and O02. These rival explanations are represented 
by the threats to internal validity (Campbell and Stanley, 
1966) and by the particular techniques by which data are 
collected (Denzin, 1970; Phillips, 1971). 

Internal validity deals with the issue as to 
whether reported findings are due to the effect of an 
experimental variable or due to other causal factors for which 
no accounting was made in the design of the research. Some 
threats to internal validity deal with spurious conclusions 
based upon observations made before and after the effect of 
an intervening experimental variable. Others deal with 
erroneous conclusions based upon supposed differences between 
comparison groups. 

The threats to internal validity are (Campbell and 
Seantey, 1966): 

=P sctory effects. 
- Subject maturation. 
=eRe-Lest Sophistication. 
- Changes in instrument calibration. 
= Stawistical regression. 
- Bias in selection of subjects. 
—- Differential subject drop-out effect. 
- Interaction of selection bias and maturation effect. 
The content and affect of each of these threats are explained 


in detail in Chapter III. 
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The techniques by which a researcher collects data 
can have a Similar effect on his or her conclusions. A study 
might report an observed level of user satisfaction with the 
implementation of a new system, ascribing this finding to the 
effect of experimental variable X, say a program to involve 
users in the system’s design. Yet, in reality, the observed 
difference might be caused by the very wording of the 
questions in a questionnaire used to measure satisfaction. The 
order in which the questions appear in a questionnaire or the 
graphic format of the questionnaire itself could also create 
the observed effect. 

The difficulty with the ways in which researchers 
collect data is, according to Phillips (1971) that. ee 
measurements of most variables are caused less by the ‘true’ 
values of those variables than by the effects of measuring 
them." If the three conditions of covariance, time order and 
the exclusion of rival explanations are met, then a researcher 
can confirm that a Concept X is the cause of a Concept Y cueatets 
the study of system effectiveness can establish causal 
inference by generating causal propositions, then it has the 
foundation for constructing a formal theory of system 
effectiveness. 

If students of information systems can construct 
a formal theory of system effectiveness, then the discipline 
of MIS will be able to describe, explain and predict the 
conditions under which an implementation of an information 
System is likely to be a success or a failure. If an academic 
discipline is not engaged in the construction of formal 
theories that describe, explain and predict, what then is it 
doing. If a field of inquiry is not engaged in doing science, 


it risks being condemned to impotency and irrelevancy. 
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II. LITERATURE REVIEW 


A. COST-BENEFIT ANALYSIS 

Bemaings on whether or not Cost-Benefit Amalysirs ghould 
be used, the problems and weaknesses associated with using it 
and the methods by which it should be conducted are summarized 
below. 

ves, et.al (1983), States that: 


Theoretically, the determination of information system 
weeme 15 a Matter of economics: the costs of system 
Operation and development are subtracted from the actual 
benefits (in organizational effectiveness) to obtain the 
net value of the system to the organization. 


iimoractice: 1). intangible costs and especially benefits 
Gm thie information systems are difficult to recognize and 
Monconvert to their monetary equivalent; 2). Some DSS are 
Meea fOr disparate, relatively unStruetured, ad hoc 
Ge@erswons; objectively assessing the benefits of such 
systems may be nearly impossible; 3). Data on system 
success may be determinable but not recorded by the 
Organization and, therefore, unavailable for research 
purposes. 


meen larch 1981), suggest that traditional cost-benefit 
g@ealysis is not well-suited to Decision Support System. 
However, he notes that it is effective for many computer-based 
Systems. Cost-benefit analysis 1s sensitive to assumptions 
such as discount rates and residual value. 

If cost-benefit analysis is appropriate or can be applied, 
there are problems or weaknesses. Chervany and Dickson 
Mero; note that, asmajor source of difficulty is the failure 
to view Management Information System development and evalua- 


—]l0n as a problem in decision-making. Keen (1981) identifies, 
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a general weakness of the cost-benefit approach as being its 
requirement for knowledge, accuracy and confidence about 
issues which for innovations are unknown, ill-defined and 
uncertain. The complex calculations of cost-benefit analysis 
are replaced in value analysis by rather simple questions 
about usefulness. 

King and Schrems (1978), highlight three problems with 
using cost-benefit analysis for evaluating decisions about 
information system in organizations. First, the difficulties 
of assigning cost and benefits, second, the failure to specify 
the critical characteristics demanded of the system and 
finally, social and political realities. 

There are several methods suggested in the literature for 
conducting cost-benefit analysis; the most common methods are 
discussed in the following paragraph. Chervany and Dickson 
(2970), "conclude that: 


It is the prime responsibility of the MIS specialist to 
synthesize these three elements - controllable variables, 
constraints, and uncontrollable variables - and project 
payoff functions (costs and benefits) for the alternative 
MIS configurations under consideration. 


They further note that cost-benefit analysis are the 
results of decisions, of not time; therefore, techniques of 
economic evaluation of management information systems must be 


decision-activity oriented. 


B. USER SATISFACTION 

This section summarizes comments and observations from 
several authors on user satisfaction. First, a discussion on 
productivity gains that can be addressed by Management 


Information System is presented. Next, effectiveness and 
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efficiency for users information system are discussed. 
Finally, some of the expected payoffs and costs are des- 
cribed. 

Chandler (1982) states: 

IS can be evaluated from two different perspectives: 


°* Focus on the computer system domain, where performance 
is measured in terms of resource utilization, cost and 
efficiency. 


e Focus on user domain throughput, where reliability and 
response time are common measures. 


An information system can be viewed as a symbiotic 
relationship between the users of the system and the system 
itself. Ideally, an information system should be evaluated 
with equal consideration given to both user constraints and 


to system constraints. 


1. Productivity 

Amerrezvani (1986) said, “the recent explosion of 
information technology has loosed a Pandora’s box of questions 
about how computers and electronic communications can be used 
effectively in the workplace, however, current uses of PCs 
don’t always contribute markedly to productivity". 

Olson and Lucas (1982) point out that, “Office 
Automation is expected to increase organizational productivity 
through redefinition of office work rather than increase 
Petlcmeney sot current office functions." Additionally they 
state that, "Automated office systems can provide a powerful 
mechanism for increasing productivity and improving the 
quality of work life by changing the fundamental nature of 
Organizational information processing." 


Finally, Strassmann (1985) concludes that: 


is 


The cost of automating office is a matter of people not 
a matter of machines. OA projects are dominated by user 
costs, or organization cost. Due to heavy initial orga- 
nizational costs, the benefits from most investment in IT 
can be realized in the long run. Realizing a long-term 
payoff from IT depends on the gain one obtains from people 
who learn how to work more effectively. 


The primary objective of such projects is to rapidly make 
people much more productive. And do not automate 
obsolete patterns of work, because you will not get the 
full benefit of the technology. You will only get minor 
gains and minor gains soon will no longer be sufficient 
to maintain a competitive edge. 


2. Effectiveness/Efficiency 
Ives, et.al. (1983), state that: 


DSS can be judged on two criteria: efficiency and 


effectiveness. Efficiency deals with how well they 
do what they do. While effectiveness takes a broader 
FOCUS. Are they doing the right thing? Are new 


computer technologies being successfully integrated 
into the organization? 


Strassmann in (1985) "Making Office Systems Pay Off" 
(1985), prefers to measure effectiveness - doing the right 
thing. Effectiveness relates to the operation of groups, not 
rmGaviduale and effectiveness 1s judged by customer 
Satisfaction. A higher level of payoffs come from improving 


group effectiveness and efficiency. 


3. Payoffs/Costs 
Strassmann (1985), states that: 


The goal of office automation from the point of view 
of many top managements, is to reduce employee head- 


count or to reduce direct expenses. These expenses 
include telephone charges, courier service, travel, 
etc. Management wants bottom line payoffs. Office 


automation can no longer be justified by promoting 
the use of technology with the vague claim that it 
will “increase productivity." 
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OA", identifies six cost and payoff characteristics associated 
with Office Automation (OA). The prime responsibility of 


management on information technology projects is to cope with 


Levels of payoffs, There appears to be a progression 
of potential payoffs from installing office automation 
systems. The lowest level payoffs come from 
automating individual tasks, aimed at reducing head- 
count. These payoffs probably will not be great 
because they involve individual tasks and individual 
employees. A higher level of payoffs come from 
improving group effectiveness and efficiency. 


Getting a financial payoff from automated office 
systems will not be easy. New technology, by itself, 
is not the answer. The big payoffs will come from 
solving the "people" problem. 


Additionally, Strassmann in (1985) "The Real Cost of 


Organizational issues rather than with technology issues. 


Organized labor is becoming keenly aware of the negotia- 
ting leverage that mismanaged OA projects hand over to 
them. 


The cost of any activity that is labor intensive tends 
to rise faster than the general rate of inflation. 


Sending employees to classes on company time at a 
company training center is the most expensive way of 
learning. 


Most importantly, the risks of run-away organizational 
costs due to technical system failures cannot be 
tolerated. 


To create the right environment for sharing organization 
learning, it will be necessary to create new standards 


for access to information sources. 


Until we have better insights into the behavioral 
aspects of people, organizational costs will be high. 
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Olson and Lucas (1982), state that: 


The political view assumes that automation implies a 
Givision of labor that leads to decrease in skill, 
knowledge, and worker control over activities. 


C. OTHER MODELS 

This section discusses other models or perceptions 
collected in the nonempirical literature review. They are 
broken-down into pieces that address productivity, effecti- 


veness/efficiency and other general perspective. 


1. Productivity 
Borko (1983) defines productivity as the ratio of the 
number of units of output to the number of units of input. 
Productivity can be measured by converting input and output 
units to their respective dollar equivalents. He further 
States that: : 


Productivity rises when the number of units of output 
increases while input holds steady; or conversely, when 
the number of units of output remains fixed and the number 
of units of input decreased. 


Obstacles to measuring knowledge worker productivity: 


° Defining the nature, the value, and the unit of measure 
for the output of the knowledge worker. 

* Tendency to deal with only the quantifiable and not with 
the quality of the outputs. 

* The work done by a knowledge worker in a current time 
period may not show results until several periods later. 


Carlyle (1987), states, "The difficulty of measuring 
productivity gains has also deterred potential buyers, but 
some vendors, like Texas Instruments, are in it for the long 
have He further notes that, at a time when the industry 


18 paying lip service to the need for Computer Aided Software 
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Engineering (CASE), computerized tools for software developers 
are emerging ahead of an accepted productivity index or 
effective measure. Many large companies are afraid that they 
could invest more in trying to improve productivity than they 
would actually be able to save. 

Mason, et.al. (1984) said, "In the area of public 
productivity and computer use, computers serve three purposes; 
reducing costs...increasing revenues...most impor — 


tant...providing better services." 


2. Effectiveness/Efficiency 

Hamilton and Chervany (1981), contend that, effec- 
tiveness is determined by comparing performance to objectives. 
The way to assess system effectiveness is to: 

* Determine the task objective of the system or the 
organizational units [using] the system and to develop 
Criterion measures to assess how well the objectives 
are being achieved. 

System effectiveness is determined by attainment of a 
normative state, e.g., a standard for "good" practices. 
Effectiveness is conceptualized in terms of resource viability 
rather than in terms of specific task objectives. Evaluation 
of system effectiveness is difficult due to its multi- 
dimensionality, its quantitative and qualitative aspects and 
the multiple, and often conflicting, evaluator viewpoints. 

The assessments of management information system 
effectiveness are often controversial and can be sources of 
disagreement and conflict between different functional groups 
involved in MIS implementation: users, management information 
system developers, internal auditors, and top management 
personnel. Maggiolini, (1986), identifies four points that 


can improve office automation technology efficiency by: 
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* Automating all or some parts of the office information 


process; 
¢ Eliminating some of the transformations of medium; 


¢ Eliminating or reducing the shadow activities/functions; 


* Speeding up the information process itself. 


3. General Perspectives 
Hamilton and Ives (1982), said that, "the key to good 
research, though is not just in choosing the right research 
strategy, but in asking the right questions and picking the 
most powerful method(s) for answering questions given the 
objectives, research setting and other salient factors." 
Ginzberg (1978), states that: The key point measure of 
effectiveness depends upon the intent of a project its goal. 
We can develop a behavioral measure of Operation 
Research/Effectiveness if we explicitly address the issue of 
goals at the start of the project. Finally, Strassmann 
(1976), suggests nine steps to better management of Management 
Information System: 
a. Identify costs 
Keep scores on unit costs 
Establish standard costing 
Set up accountability centers 
Apply competitive pricing 
Plan for the long term 


Let the users control 
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Deemphasize the technology 


r> 


Use job enlargement 
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III. METHODOLOGY 


SELECTING STUDIES OF SYSTEM EFFECTIVENESS 


Choosing a Medium from Which to Select Studies 


Published articles were chosen as the source from 


which to select the studies of system effectiveness to be 


reviewed in this’ thesis. Published transactions of 


conferences were excluded for three reasons: 


ar. 


The difficulty of obtaining comprehensive bibliographical 
search coverage of all conferences that hold sessions 
related to system effectiveness. For example, an on-line 
bibliographic search oof computer and management 
information system (MIS) failed to cover the important 
annual International Conference on Information Systems. 


No evidence exists to support a decision that papers 
presented at a conference that publishes its transactions 
are superior to a conference that does not. Leaving 
aside the problems of obtaining reliable bibliographic 
coverage of conferences that do publish transactions, to 
include them would unfairly exclude papers from non- 
publishing conferences. 


Both of the preceding issues are moot in that presenting 
a paper at a conference is, by convention, a step to 
submitting it for publication by a journal. Indeed, 
papers accepted by the editors of the International 
Conference on Information Systems convey the best of them 
to journal editors for consideration as published pieces 
(Gallegos,1985). In short, conference papers worthy of 
publication will appear in journal omitting conference 
papers prevents redundant entries. 
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2. Choosing Journals from Which to Select Studies 

This review of academic studies began with a selection 
of journals from which reports of these studies would be 
taken. All reports of studies of system effectiveness in 
English-language journals in the field of management 
information systems (MIS) and the related field of management 
science were surveyed. We took as an authority for choosing 
this set of journals the works of Davis (1980), Hamilton and 
Ives (1983) and Vogel and Wetherbe (1984). A consensus of 


journal names from these studies yields the following list: 


Academy of Management Journal 
Accounting Review 

ACM Computing Surveys* 
Administrative Science Quarterly 


Communications of the ACM 
Data Base 


Datamation 


- Q mo Aaa DF 


Decision Sciences 


f+ 


EDP Analyzer** 

Harvard Business Review 

TEEE omputer === 

Information and Management 

Interfaces 

Journal of Management Information Systems 
Journal of Systems Management 

Management Information Systems Quarterly 
Management Science 

Omega 

s. Sloan Management Review 


* ACM: Association for Computing Machinery 
xx EDP: Electronic Data Processing 
x*x* TEEE: Institute of Electronic and Electrical Engineering 
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Hereafter, this list is known as the consensus list. To it, 
was added Public Productivity Review which was discovered 
during a search for periodicals related to the issue of 
productivity. While the preferred publication outlets on the 
consensus list are refereed journals written by and for 
academic members of the MIS community, the list includes 
periodicals that are aimed at an audience of MIS practitioners 
and some that are not Journals per _se but are commercially 


published trade publications. 


3. Choosing a Time Span from Which to Select Studies 

The work on this review began in January, 1989 and was 
completed by June, 1989. The last journal articles that were 
considered for inclusion were published with the calendar 
period of 1988. 1970 was chosen as the beginning point for 
on-line and manual bibliographic searches. 

However, no time limit was placed on inclusion of 
Studies that were discovered by virtue of references in 
articles found in the 1970 to 1988 range by manual or on-line 
searches. The assumption was that articles worthy of 
inclusion that were published prior to 1970 would be 


referenced in articles published after 1970. 
4. Searching for Studies of System Effectiveness 


a. On-Line Bibliographic Search 
In January, 1989, the Dialogueue access station in 
the Dudley Know Library of the Naval Postgraduate School was 
used for a bibliographic search based on key words in the 
titles and abstracts of articles. The following search key 


was constructed: 
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(Productivity or Effectiveness) and (Measures or 
Measurement or Assess or Assessment) and Information 
Systems. 

This search identified 154 items that matched the 
search key. Abstracts on the 128 of these that were published 
in English were requested. Out of that set, 51 items were 
identified that dealt with the productivity or effectiveness 
of information systems. The other 77 items dealt with the 
monitoring of computer performance, productivity of software 
development or topics unrelated to system effectiveness but 
nevertheless included some combination of the search key words 
in their titles or abstracts. 

Of the 51 items related to system effectiveness, 
26 were conference papers. Out of the 25 items dealing with 
productivity or effectiveness of information systems that were 
journal papers, ten were in journals on the consensus list. 
This search covered several important conferences related to 
information systems, including some of which we would not 
otherwise have known. 

Nonetheless, as mentioned earlier, the flagship MIS 
conference, the annual International Conference on Information 
systems, was not represented. In retrospect, this on-line 
search was naive in its conception and incompetent in its 
execution. During the manual bibliographic search, it was 
discovered that the "productivity" and "effectiveness" 
keywords were scarcely comprehensive in the coverage of the 
literature of system effectiveness. 

The keywords that authors in this literature 
applied to their works included: 

=a COS 


- Benefit 
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- Cost Benefit 

- Evaluation 

- Success 

- Utilization 

- Performance 
Moreover, the search key applied only to Dialogue file 13, 
which is the INSPEC data base for the period from 1977 to 
1988. That this file began its coverage at 1977 rather than 
1970 already meant that the original intention was 
compromised. A different INSPEC file contained items 
published before 1977. That file would have been included in 
the search but for a funding limit on access to bibliographic 
search services. 

Subsequent experimentation with on-line 
bibliographic searching revealed that other databases in the 
Dialogue set might have yielded more articles. Certainly, by 
not searching beyond the INSPEC file, which covers the 
computer and MIS fields, publications in fields such as 
general management (Harvard Business Review, Sloan Management 
Review) and management science (Management Science, Decision 
Science, Administrative Science Quarterly) were missed. 

Because limited funding prevented further on-line 
bibliographic searching with a wider net of key words applied 
a broader set of databases, this study turned to manual 


searching. 


Bb. Manual Bibliographic Search 
We manually examined the Periodicals on the 
consensus list for the years 1970 to 1988 were manually 
examined (although some of these periodicals were founded 


after 1970). Indices of article titles for the covered years 
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were examined or every issue of a periodical (where indices 
were not available) were examined. The manual search sought 
titles of articles that seemed to deal with the effectiveness 
of information systems. This included articles on the 
performance of systems in specific industries or by type of 
specific application (such as decision support systems) as 
well as information systems in general. This effort yielded 
93 articles purporting to deal with the measured effectiveness 
of information systems. 

The manual search, not surprisingly, found the same 
ten titles that were revealed by the on-line search. The 
number of articles identified in the on-line search compared 
to the far greater number turned up by the manual effort was 
dismaying. This was the point at which it was discovered how 
hopelessly limited had been the key word synonyms for 


productivity and effectiveness that were used. 
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c. References from Journal Articles 

Reference lists at the end of articles found 
manually were used to track down important works that either 
were published before 1970 or whose relevance was disguised 
by its titles. This procedure produced an additional 31 
articles. These included works that dealt with system 
effectiveness as well as articles that dealt with 
pPreduct 13 eye cost-benefit analysis, organizational 
effectiveness and research methods that were written for a 


broader audience than the MIS discipline. 
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d. Gathering of Article Reference by Other Means 
When studying a particular topic, a researcher 
becomes sensitized to its appearance in any form. Discussions 
of system effectiveness or productivity in popular or trade 
publications, that might otherwise go unnoticed, will now leap 
into a researcher’s awareness. Even colleagues who are 
peripheral to a study become sensitized as well. They offer 
references and clippings dealing with productivity and the 
effectiveness of information systems. By these processes the 
inventory of works on productivity, system effectiveness and 

the measurement of effectiveness in this study was built. 
A good deal of these were from popular and trade 
publications. They included magazine articles, newspaper 
items, student theses, books, transcripts of speeches and 


promotional literature on software and consulting services. 


5. Sorting the Works on System Effectiveness 
In combination, the preceding methods yielded an 
overall list of 118 references that was initially considered, 
by virtue of their titles, for review in this thesis. They are 


listed in Appendix A. 


a. Empirical Studies 

A reading of the 118 articles in Appendix A 
identified 45 empirical studies of system effectiveness. 
These 45 articles report empirical studies of the 
effectiveness of an information systems or an aspect of 
research techniques for studying system effectiveness. 

These articles are distinguished from others in the 
Original set of 118 in that they report findings based on data 
collected specifically for the study being reported. fThese 


are the studies that this thesis evaluates. They are studies 
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are listed in Appendix B. Among these 45 were articles that 
Gid not deal directly with a study of a system implementation 
but rather were concerned with the development of a research 
tool to study implementation success. These were retained 
because their study research techniques contributes to how we 
know what we know about system effectiveness. 
Excluded from this review were studies that dealt 
with: 
1. Computer performance. 
2. Software development. 
3. Other topics not related to information system 
effectiveness. 
b. Non-Empirical Works About the Study of System 
Effectiveness 
Excluded from this review of empirical studies were 
articles which did not report findings of a concrete study of 
a system implementation. These included essays on system 
effectiveness, literature reviews of system effectiveness or 
prescriptions for the methods of studying information system 
effectiveness. Appendix C lists the 22 articles in this 
GEOUpE They are, of course, the references that were 
discussed in the preceding chapter which reviewed the MIS 


literature on the assessment of system effectiveness. 


c. Miscellaneous Literature 
The remaining references from the initial 118 were 
categorized as: 
- Popular and trade literature. 


- Literature dealing with productivity or organizational 
effectiveness in general but not specifically related to 
MIS. 
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- Works dealing with research methodology without specific 
application to MIS. 


These are listed in Appendix D. 


B. REVIEWING THE EMPIRICAL STUDIES 
The 45 empirical studies (listed in Appendix B) were 
individually evaluated for: 
1. Design of data collection 
Handling of threats to internal validity 
Handling of threats to external validity 


2 

S 

4. Time dimension 

5. Data collection methods 
6 


herinitiomrm OL GLfrrectiveness 


1. Design of Data Collection 
From a reading of the descriptions of research 
methodology and data collection techniques in each study, it 
can be determined if it: 
a. Employed an explicit, experimental variable. 


b. Gathered data before the administration Or an 
experimental variable. 


c. Gathered data after the administration of an experimental 
variable. 


d. Gathered data from comparison groups at the same points 
in time. 


e. Whether the membership in comparison groups was 
randomized. 


Knowledge of these five elements permitted classification of 
each study according to its research design using the Campbell 


and Stanley (1966) taxonomy. 
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2. Handling Threats to Internal Validity 
Once a study was classified according to the Campbell 
and Stanley taxonomy, it was possible to assess its ability 
to handle each of the threats to internal validity listed 


below: 


a. History 

A history effect involves events occurring between 
a pretest observation and a posttest observation which affect 
an observed phenomenon along with an intervening variable X. 
Designs that collect data at both pretest and posttest points 
but lack a parallel control group cannot control for the 
effect of history on a study’s findings. Designs lacking a 
pretest point but involving a comparison group, while weak in 


other respects, can control for the effect of history. 


b. Maturation 

Maturation refers to changes within the respondents 
themselves that can account for differences in measurements 
between the first and second observations. As with the history 
effect, designs that collect data at both pretest and posttest 
points but lack a control group cannot control for the effect 
of maturation. Where a design lacks a pretest point but 
involves a comparison group, the capacity of the design to 
exclude maturation as a rival explanation of its findings in 
not clear from a design alone (as in the case of the static 
group design). Designs that use randomized comparison groups 


(such as the posttest only control group) are not affected. 
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c. Re-test Sophistication 

Re-test sophistication arises when respondents 
become skilled at taking a test, filling out a questionnaire, 
answering in an interview or taking part in an experiment. As 
much as the intervening variable, such changes in skill can 
cause changes in readings at a second observation. Where a 
design lacks a pretest phase, yet uses comparison groups, this 
effect is controlled. Designs that involve a pretest and 
posttest but lack a comparison group (Such as the one group 
pretest/posttest design) are threatened by re-test 
sophistication, For other designs, the effect is neither 


controlled nor threatening. 


d. Instrument Calibration 

Where people are the instruments of recording and 
measuring, changes within them can produce changes in the in 
reports of events. Examples of human re-calibration are 
fatigue, growing skill at conducting interviews, boredom or 
developing self confidence as observers. This effect is 
controlled in designs that involve comparison groups with 
randomized assignment of members or, lacking that, have no 
pretest stage. The one-group pretest/posttest design is 


vulnerable to instrument calibration effects. 


e. Statistical Regression 
| Where comparison groups in a study of system 
implementation have been selected precisely for their extreme 
differences in scores on tests or surveys before a study, the 
Statistical regression toward the means score is likely to 
occur on the posttest observation. Generally, respondents with 
extreme scores on one occasion will not produce the same 


extremes on another occasion. A design alone does not tell 
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whether statistical regression affects a particular instance 
of a one-group pretest/posttest design. The static group 
design, however, is vulnerable to is effect. Designs involving 
randomized comparison groups are able to control for the 


effect of regression. 


£. Selection Bias 

Selection bias is a rival explanation for a study’s 
findings when comparison groups assumed to be equivalent are 
actually different due to biases operating during the 
selection of respondents. These differences can account for 
the differences in reported data between the two groups. Any 
design with a pretest and a posttest can control for this 
effect. Those lacking a pretest or a randomized comparison 


group are vulnerable to selection bias. 


g. Respondent Mortality 
Between non-randomized -comparison groups, there 
can be a selective drop-out of participants from a study while 
researchers assume that dropping-out is random. This 
differential drop-out effect can account for differences in 
data reported from the groups. The designs that control for 
this effect and those that are vulnerable to it are identical 


to that of the effect of selection bias. 


h. Selection-Maturation and Other Interactions 
In multi-group research designs, the interaction 
of a selection bias with a maturation effect could be mistaken 


for the effect of an intervening experimental variable. 
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3. Handling Threats to External Validity 
As with the threats to internal validity, knowledge 
of a study’s placement in the Campbell and Stanley (1966) 
taxonomy of research designs allowed researchers to assess 


its capacity to exclude threats to external validity. 


a. Interaction of Testing and an X Variable 

In certain kinds of studies, the experience of 
participating in an experiment, completing a questionnaire, 
submitting to an interview or being observed at pretest stage 
sensitizes a group of respondents to an X variable. That is, 
by being subjects ina study, these people have a heightened 
sensitivity to an X stimulus that is not characteristic of the 
general population. 

To that extent, the findings of such a study are 
not generalizable beyond the sample group. The one group 
pretest/posttest design and the true experimental design are 
both vulnerable to this effect. Groups that have some elements 
of non-pretest comparison such as the posttest only control 
group and the Solomon four-group design are able to control 


for it. Other designs are not affected either way. 


b. Interaction of Selection and the X Variable 
Campbell and Stanley (1966) use as an example of 
this threat a researcher seeking a school district to provide 
a sample of students for a study. Nine districts turn down the 
researcher. A tenth district agrees to participate. Surely, 
in some way, the tenth differs from the other nine. The 
results that this study yields will, therefore, not be 

generalizable to a large population of students per se. 
To the extent that an experimental variable in this 


study has differential effects between an experimental group 
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and a control group, it is likely that characteristics of the 
school district caused the X variable to be more successful 


than it would begin other school districts. 


c. Reactive Arrangements 
The artificiality of an experimental setting or of 
questionnaire choices put respondents in a Guinea-pig, I-am- 
special role. Whatever behavior that is thereby observed or 
recorded does not represent how these same people are likely 


to act in everyday life. 


d. Multiple X Interference 

While a conventional research design involves a 
single administration of an X variable to experimental group, 
some quaSi-experimental designs such as the equivalent time 
Samples design, and the equivalent materials samples design, 
involve repeated administrations of an X variable to one group 
of respondents. ‘ 

The trouble is that the effects of prior 
administrations remain with the respondents, building up with 
each subsequent administration. Such groups undergo an 
experience as research subjects that has few counterparts in 


everyday life. Thereby, they are rendered unrepresentative of 


the larger population. 


4. Time Dimension 
The Campbell and Stanley (1966) taxonomy of research 
designs also provides a means to assess whether a study has 
taken account of the time dimension as a basis’ for 
establishing causal inference. 
Studies employing pretest and posttest observations 


and those that employ time series multiple observations, by 
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definition, account for the time dimension. Studies involving 


an observation at a single point in time do not. 


5. Techniques of Data Collection 
By convention, an empirical study is obliged to 
describe the technique by which it gathers data. That 
convention was upheld by all 45 of the studies reviewed here. 
Each study was classified according to the following taxonomy 
of data techniques: 


a. Self-completion questionnaire, administered under the 
supervision of the researchers. 


b. Self completion questionnaire, mailed out to subjects to 
be returned by mail. 


Interview questionnaire. 
Telephone interview. 
Laboratory experiment. 


Non-obtrusive technique: archival records. 


Q ho a A 


Review of observations by expert panel. 


6. Definition of System Effectiveness 
Each study was read for its definition of how a system 
would be judged to be effective. No taxonomy of such 
definitions exists a priori The classification categories 


emerged in the process of reviewing the studies. 


7. Degree of Change in Effectiveness Reported in the 
System 

= This thesis suggests that the appropriate enterprise 

for academic research in MIS effectiveness is the construction 

of a formal theory of system effectiveness. A theory of system 

effectiveness necessarily involves a knowledge of the 

determinants of such effectiveness. To understand the 


determinants of effectiveness, a researcher must know by how 
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much the effectiveness of an organization was improved or 
degraded by the implementation of an information system. 

Therefore, it is appropriate to inquire as to the 
level of changes in effectiveness that are reported in each 
empirical study of system effectiveness. If doubts about the 
success of information systems are harbored by senior line 
Managers, then a knowledge of the reality of information 
payoffs is a contributes to the current discussion. What do 
these 45 academic studies say about the successful application 
of information systems? 

Each study was reviewed for indications of the level 


of reported changes in system effectiveness. 
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Iv. FINDINGS 


Appendix E contains an annotated description of each of 
the 45 studies evaluated here. Appendix F contains the 
detailed tables of findings for each aspect of the studies. 
The studies are listed in chronological order by date of 


publication. 


A. DESIGN OF DATA COLLECTION 
In Table F-1 (Appendix F), the number at designations at 
the top of the columns refer to evaluation elements in 
assigning each study to a design category according to the 
Campbell and Stanley (1966) taxonomy as follows: 
1. The study collected data in a pretest stage. 
2. The study employed a comparison group. 
3. Respondents in the study were assigned to comparison 
groups on a randomized basis. 


1. Non-Experimental Designs 


a. One-Shot Case Study 
Table 1 distills the distribution of research 
designs in these 45 studies out of the detail of Table F-1 
(Appendix F). Table 1 shows that the one-shot case study is 
the research design of choice in MIS studies of system 
effectiveness, by a two to one margin. A one-shot case study 
has no explicit administration of an intervening, experimental 


variable (X). Rather, as the brackets around the X in the 
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design notation indicate, the subjects from whom data was 


collected, at the sole observation point 0O,, were assumed to 


TABLE 1 
DISTRIBUTION OF RESEARCH DESIGNS‘ 
This Design 


as a 
Campbell Percentage 
& Stanley of Studies 
Research Design: Design Notation Reviewed ;: 
One Shot Case Study 66.7% 
[X] 0, 
Static Group Comparison 17.8% 
Xx fe) 
0 
One Group Pretest/Posttest 4.4% 
Oo, xX oO: 
Experiment 4.4% 
R e) Xx e) 
R e) e) 
Posttest Only Control Group 4.4% 
R Xx Oo 
R e) 
Time Series Cees 


O O x O O 


"These percentages sum to 99.9% due to rounding errors. 


have been exposed to the effects of a non-existent 
experimental variable. In place of explicit exposure to an 
X experimental variable, the subjects at 0O, are asked to 
recall something. Human recall is notoriously unreliable 


(Haberman, 1963). For that reason, the one-shot case study 
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is "the weakest form ofsurvey design" (Denzin, 1970). 
Fundamental to the gathering of scientific evidence is the 
need for recording differences (Campbell and Stanley, 1966). 
Scientific testing requires the making of comparisons. No 
comparisons are made in one-shot studies. While one-shot 
studies often involve survey instruments that describe their 
face validity and Cronbach reliability indices in detail, all 
of that is a case of misplaced precision. 

It is another case of shuffling the psychometric 
deck chairs on the methodological Andrea Doria. In the 
estimate of Campbell and Stanley, rival explanations for the 
findings reported by one-shot studies are so numerous as to 


render them nearly useless in establishing causal inference. 


b. Static Group Comparison 

As 1s shown in Table 1, the next most frequent 
design in MIS research on systems effectiveness is the static 
group comparison. This design offers the virtue of some basis 
for examining differences. The difficulty, however, is in 
attributing the differences to exposure to an experimental 
variable by one of the groups. That is because membership 
in the comparison groups in this design is not a randomized 
assignment. 

The findings are vulnerable to the - rival 
explanation that differences between the groups predated the 
intervention of an X variable. Where the comparison groups 
are deliberately selected for their extreme differences, as 
they were in some of the reviewed studies, the rival 
explanation gains potency. Another problem with the static 
group comparison is the possibility of biased drop-out by 


respondents in the period after the X variable intervened. 
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c. One Group Pretest/Prosttest 

The one-group design has the virtue of a deliberate 
attempt to measure the differences between a group observed 
at O, and at O,. Unfortunately, its lack of a comparison group 
leaves it subject to a distracting set of rival explanations 
fer its findings. Chief among them are other effects 
transpiring between O, and O, besides the X variable. 

While it is superior to the one-shot case study, 
the only recognition the one-group design merited from 
Campbell and Stanley was that it was worth doing only where 
nothing better could be done. They employ the one-group as 
a bad example of research design that deludes researchers and 
their constituents alike by its closeness to a true 


experimental design. 
2. Experimental Designs 


a. True Experiment 
The true experiment, having pretest and posttest 
data collection points paralleled by randomized comparison 
groups, controls for all threats to internal validity. The 
difficult with the MIS empirical studies of effectiveness is 


that only four percent of them used this design. 


b. Posttest Only Control Group 
While it lacks a pretest data collection stage, 
the posttest only design nonetheless qualifies as_ an 
experimental design by virtue of its randomized comparison 
groups. As with the true experiment, its difficulty is the 


rarity with which it was practiced in MIS research. 
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3. Quasi-Experimental Design 
A quasi-experimental design, a description coined by 
Campbell and Stanley, is an attempt to approximate 
experimental design in a field setting by scheduling the whom 
and when of measurement even though the experimental variable 


itself is not amenable to scheduling. 


a. Time Series 

While it might appear that a time series design 
shared the vulnerabilities of the one-group pretest/posttest, 
this is not the case. The repeated measurements before and 
after an experimental variable controls for threats to 
internal validity such as maturation, re-test sophistication, 
instrument calibration, statistical regression and _ the 
interaction of selection bias and maturation. This renders 
it a much sounder design that the one-group. Regrettably, 


only 2% of the reviewed studies employed a time-series design. 


B. HANDLING THREATS TO INTERNAL VALIDITY 

The detailed evaluations of the handling of threats to 
internal validity by each of the 45 studies reviewed are shown 
in Table F-2 in Appendix F. A +1 in a column means that a 
study controlled the threat to internal validity represented 
by the column heading. A -1 indicates that a study was 
vulnerable to that particular threat. A question mark means 
that the effect of the threat upon this study cannot be 
determined solely by knowledge of the research design. No 
symbol indicates that a threat has no effect on the study in 
whose row it occurs. 

Recall that internal validity asks whether reported 
findings are due to the effect of an experimental variable or 


result from causal factors not accounted for by the research. 
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Where a study can control threats to internal validity, its 


findings resist challenges to their scientific standing. 


TABLE 2 
THREATS TO INTERNAL VALIDITY 


Effect Effect Effect 
to Vulner- Cont- Not No 
able rolled clear effect 
HIStory 713% 21% 
Maturation 73% 9% 18% 
Testing 4% 29% 69% 
Instrumentation 4% 27% 2% 67% 
Regression 29% 4% 67% 
Selection 84% 16% 
Mortality 84% 16% 
Interaction of 
Selection and 22% 11% 67% 


Maturation 


To the extent that a study cannot exclude the rival 
explanations for its findings posed by threats to internal 
validity, the scientific meaning of the reported data will be 
lost. The evaluations in Table F-2 are summarized in Table 
2. It reveals that, on average, slightly over 73% of the MIS 
studies of system effectiveness are vulnerable to the 
likelihood that their findinas can alternatively he accounted 
for by the effects of history, maturation, selection bias and 
MOrcality. 

This is a consequence of the fact that two-thirds of the 


MIS studies reviewed here employ a one-shot case study design. 
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No studies were clearly vulnerable to statistical regression 
aS a rival explanation. Moreover, rather few (4%) had their 
internal validity threatened by re-test sophistication or 
instrument calibration. 

The picture that emerges here is that of an academic 
discipline that cannot demonstrate with certainty that its 
empirical findings mean what they say. These findings erode 
a basis for establishing causal inference in the construction 
of a formal theory of the effective implementation of 


information systems. 


C. HANDLING THREATS TO EXTERNAL VALIDITY 

Table F-3 in Appendix F contains a detailed evaluation of 
the handling of threats to external validity by each of the 
45 studies reviewed in this thesis. The column headings at 
the top of Table F-3 (to the right of the year column) 
includes the two external validity threats arising from 
interactions between the experimental variable and 
characteristics of a study’s design. 

tier iivst of these is Che intemaction of re testungwand 
an X variable. The second is the interaction of selection 
bias and an X variable. The meaning of both of these are 
explained in Chapter III. The third column is the threat from 
reactive arrangements. The fourth is the threat from effect 
oe multiple exposures to an X variable. The meaning of the 
+1, -1, ? symbols the same as in Table F-2. 

Where internal validity asks whether the findings of a 
study can be trusted to mean what they claim, external 
validity deals with the extent to which the findings of a 
study are generalizable beyond the study to a larger 


population. 
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TABLE 3 
HANDLING OF THREATS TO EXTERNAL VALIDITY 
Effect to Effect Effect 





Vulner- Cont- Not No 
Threat able rolled clear effect 
Interaction of Re-Testing 
with X Variable 11% 4% 85% 
Interaction of Selection 
Bias with X Variable 89% 11% 
Reactive Arrangements 16% 84% 


Interference of Multiple 
Administrations of 100% 
X Variable 


Table 3 summarizes the detailed evaluations of threats 
to external validity for each of the 45 empirical studies. 
The use of research designs in MIS empirical assessments 
determines these findings. With 11% of the reviewed studies 
employing a pretest stage, exactly 11% are found to have their 
generalizability threatened by a re-test effect. None of the 
reviewed studies involved repeated administrations of an X 
variable; therefore, none were vulnerable to multiple xX 
interference. 

However, the 89% of the studies that used non-experimental 
designs (chiefly one-shot case studies) precisely account for 
the vulnerability to an interaction of selection bias and an 
X variable. External validity deals less with a fundamental 
question of scientific credibility than it does with a simple 
question of relevance. Every study reviewed here is 
vulnerable to a challenge to the generalizability of its 


findings to a larger population. Only the two posttest-only 
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studies provided a positive control over any of the external 
validity threats. 

If the findings of MIS research are not representative of 
any population beyond the studies themselves, the discipline 
is flirting with the likelihood that its core work is 


irrelevant beyond the pursuit of publication and tenure. 


D. TIME DIMENSION 

Table F-4 in Appendix F categorizes each of the 45 studies 
Simply by whether its research design permitted an accounting 
of the time dimension in the collection of its data. 11% of 
the studies accounted for time, through designs such as the 
true experiment, one-group pretest/posttest and time series. 
The other 89% of the studies employ designs that collect data 
only at a single point. Most of those are of the one-shot 
case design. The capacity of a research effort to establish 
causal inference is impaired without a way to show that 


variable X occurs before variable Y. 


E. DATA COLLECTION METHODS 

Table F-5 (Appendix F) lists the type of data collection 
technique employed by each study. Two columns of methods are 
shown in Table F-5. One third of the studies in this review 
used two methods. 

In some cases, two methods bolstered each other in an 
effort at triangulation. This happened where a survey 
questionnaire was used in conjunction with system performance 
data collected by the information system itself. In other 
instances, a first method was used as pilot study, usually for 


instrument development. 
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TABLE 4 
DATA COLLECTION TECHNIQUES FOR PRIMARY METHOD 
Asking Questions 73.3% 


Questionnaire 60.0% 
(Including mail out- 
mail back surveys: 22%) 


Interview 11.1% 
Phone survey ZS 
Not Asking Questions 26.6% 
Non-obtrusive methods 17.8% 
(Archival) 
Laboratory experiment 4.4% 
Diary 2.2% 
Panel 2.2% 
TABLE 5 
DATA COLLECTION TECHNIQUES FOR SECONDARY METHOD 
Asking Questions 80.1% 
Questionnaire 66.7% 


(Including mail eur 
mail back surveys: 27%) 


Interview 6.7% 
Semantic Differential 6.7% 

Not Asking Questions 20.0% 
Non-obtrusive methods 13.3% 
(Archival) 

Simulation Cw, = 
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TABLE 6 
DISTRIBUTION OF METHODS BY RESEARCH DESIGN 


Type of Research Design True Post 


One- Static One Experi- Test Time 
Shot Group Group ment Only Sexes 
Method 
Questionnaire 61.3% 71.4% 50.0% 100% 
incl Mailed Out) 
Mailed Out 29.0% 50.0% 
Interview RYE 14.3% 
Phone survey oy 
Archival 16.1% Jag ae > ORO 100% 
Laboratory 100% 
Diary Sues 
Panel Ses 
100%" 100% 100% 100% 100% 100% 


" Sums to 99.9% due to rounding error. 


As Table 4 shows, over 73% of the studies reviewed here use 
primary methods that involve asking people questions whether 
by self-administered questionnaire (completed in group 
sessions or in a mailed-out/mailed back mode), personal 
interview or by phone interview. 

The problem posed by this proportion is that the focus of 
inquiry is the effectiveness of systems, not of individuals. 
Even if the objection is made that systems, after all, are 
essentially comprised of people, then the fallacy of 
reductionism is in play. It is a fallacy to study a level of 


organization by reducing it to its constituents elements. If 
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reductionism were a valid approach to studying the performance 
of organizations or systems, why stop at individuals? 
Individuals, after all, are comprised of living organs. By 
this logic, we would study information systems by studying 
intestines. 

One can follow this absurd path to molecular biology and 
beyond. While individuals together comprise systems (or 
groups or teams or organizations), there is a qualitative 
distinction between the analysis involved in studying 
individuals and in studying systems comprised of individuals. 
Systems (as with teams, groups, families or organizations) are 
not merely the sum of the discrete individuals. 

Directing questions at individuals gathers data about 
individuals but collects little about the characteristics of 
the system. This leaves aside the even more troublesome issue 
about asking people questions even when it is individuals that 
are being studied. This preference for asking questions was 
even a bit stronger among the secondary methods. 

The core concern with the people-asking methods is that 
the methods tend to create the very data they are recording. 
Far from providing a neutral, innocuous means of gathering 
observations about events, these methods provoke responses 
that, in themselves, involve role-playing, Guinea-pig effects, 
looking good, passing a test, socially desirable responses, 
fabrication, data inflation and otherwise carrying on in ways 
that are not found in everyday interactions other than with 
organizational or social superiors. These same biasing and 
distorting factors are found in the reactive setting of 
laboratory experiments. But there they are well known. 

The trouble with the survey questionnaire is that it is 


assumed to have no biasing or distorting effects when it, in 
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Bealiuy 1S strongly Dlasing and distorting (Katz, 1942; Hyman, 
1949; Wyatt and Campbell, 1950; Parry and Crossley, 1959; 
Robins, 1963; Williams, 1964; Weiss, 1968; Webb et.al, 1966; 
Sechrest, 1968; Summers and Hammonds, 1969; Hambright, 1969; 
Denzin, 1970; Phillips, 1971). 

A final problem posed by the method of asking questions 
is that the data collected can only record the expression of 
attitudes. The asking-people methods are unable to record 
expressions of behavior. Behavior can only be observed or 
recorded by observers as archival data. While MIS researchers 
occasionally speak of dealing with "behavioral" issues or 
methods, in fact they are not dealing with behavior at all but 
attitudes. 

To use a questionnaire to ask a respondent about their 
behavior is not to gather data on behavior. Rather the 
process gathers data on a respondent’s attitude toward the 
behavior above which he or she is being asked. This is a 
distinction that seems largely lost on the MIS researchers in 
the study of system success. 

Examining the association of method with research design 
holds no surprises. The non-experimental designs relied 
heavily on the methods that involve asking questions. But 
then so did all of the true experiments. However, the other 
experimental design, the posttest only control group, employed 
only a non-asking method. So did the single quasi- 
Reocoental design. 

A stronq relationship might be expected between a weak 
design such as a one-shot case study and a troublesome method 
such as asking questions of people. Both evince a reluctance 
to commit to research efforts that involve time series or 


attention to the details of non-obtrusive methods. 
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The discussions in many of the reviewed studies echoed the 
conventional wisdom that gathering data on the costs of an 
information system is difficult enough. Attempting to gather 
data on a system’s benefits is regarded as futile. fThusly, 
these researchers justify turning to methods and designs noted 
more for their convenience than their scientific rigor. 

If one has learned only survey methods as a graduate 
student, one is unlikely to explore alternative methods as 
junior faculty member under pressure to achieve tenure. This 
is especially the case where those alternative methods and 
designs take longer and are more difficult. Career expediency 
leaves causal inference in the dust. That the non-obtrusive 
archival data method is practiced so little is surprising in 
that it is in the nature of information systems to yield 
information even about their own performance. 

Yet, archival data was employed as a primary method in 
fewer than 18% of the reviewed works (and as a secondary 
method in only 13%). As a non-obtrusive method, archival data 
offer the supreme advantage of having the least biasing or 


distorting effects. 


F. DEFINITIONS OF EFFECTIVENESS 

Table F-6 in Appendix F contains a list of the definition 
of system effectiveness gleaned from each reviewed study. 
These are combined and summarized in Table 7 below. 

The first grouping, corporate scores, represents the 
traditional measures of success in the private sector. Among 
the reviewed studies, they are a minor fragment of the 
definitions employed for information success. The next 
traditional measure of system effectiveness, efficiency (cost 


reduction in this case) was found in 13% of the studies. 
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fees a defingtionm@howmmuch under attack fromthe user 
attitude enthusiasts. 
TABLE 7 
DEFINITIONS OF EFFECTIVENESS FOR INFORMATION SYSTEMS 


Corporate score 4% 
Profit 2% 
Return on investment 2% 
Efficiency 13% 
Cost reduction 13% 
Intermediate goods 29% 
Information usage 5% 
Plan implementation 2% 
System usage 18% 
Job factors Ze 
Office automation 2% 
Effectiveness 9% 
Decision quality Ue 
Performance factors 2% 
Value Assessment 7% 
Value of information 5% 
System value 2% 
By-product 34% 
User attitude 34% 
Miscellany 2% 
Improved grammar a 


29% of the studies counted as a system output a factor 
that 1s actually a system input (described in Table IV-6 as 
an intermediate good). The mistake in these cases was to see 
the usage of information or a system as evidence of the 
system’s success. The correct question is whether results 
from the uSage, not the level of usage itself, benefit an 


organization. While there is much discussion in the MIS 
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literature on measurement about the need to focus on 
organizational effectiveness rather than conventional cost- 
cutting efficiency, only 9% of the reviewed studies employed 
effectiveness as a definition of success. 

The major definition of system effectiveness found in 
these studies was some expression of user attitudes. This 
approach holds that if people say they are happy with a system 
then the system is a success. That seems sensible enough. 

Yet, the popular and trade literature reviewed in Chapter 
I found that the payoff from information systems is either 
falling or not advancing commensurate with the investment made 
in it. A user’s attitude about a system is an entity quite 
separate from the economic contribution of a system makes to 
an organization. The assumptions in this approach take either 
of these two forms: 


1. A user is satisfied with a system that is productive. 
A user’s satisfaction 1S a ™ Surrogate measure for a 
system’ s* productavaey. 


2. Technical efficiency and economic contribution are not 
the point. At best, they are the means. User 
Satisfaction is the end. 


In the first case, the possibility exists that a user can be 
Satisfied with an unproductive system. The satisfaction of 
a user may not necessarily be based on a system’s efficiency 
but on other factors such as job-enlargement, better equipped 
work areas, learning of new skills or protection from 
redundancy. 

In any case, system efficiency and performance must be 
examined apart from the attitude of users. Tt would, Sex 
example, be interesting to examine situations in which users 
were highly satisfied with unproductive information systems. 


That interest can be pursued only if satisfaction and system 
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performance remain distinct. The second case represents a 
matter of ideology, most often among the organizational 
development group that has emerged from organizational 
psychology. The root of the uSser-attitude definition of 
effectiveness may be methodological. 

If a group is committed to the method of asking people, 
then the resulting answers are arrogated to an end. If you 
cannot measure effectiveness at the system level then it is 
expedient to dismiss it as a focus of inquiry. Can a formal 
theory of information system effectiveness be constructed on 
a foundation of attitudes about a system rather than 


measurements directly from a system itself? 


G. REPORTED CHANGE IN SYSTEM EFFECTIVENESS 

Table F-7 in Appendix F lists the level of change in 
organizational effectiveness, following the implementation of 
an information system, reported by each of the 45 empirical 
studies reviewed here. 

In a group of studies that purport in some or way other 
to report findings of an inquiry into the success of 
information system, only 20% actually did so. A serious 
question arises as to whether MIS can construct a formal 
theory of system effectiveness when changes in effectiveness 
are not reported by 80% of its empirical efforts? When MIS 
claims to be studying system effectiveness yet has no findings 
related to it, MIS is in danger of gravitating into 
irrelevancy in the estimate of its major constituents: senior 


line managers, MIS practitioners and the general public. 
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V. CONCLUSIONS 


A. PROBLEMS IN ESTABLISHING CAUSAL INFERENCE 


1. Research Designs 

Much of the concern about credible, appropriate 
research designs for the examination of the effectiveness of 
information systems has been treated in the previous chapter. 
A concern that still merits exercising is the proportion (two 
thirds) of the studies in this review that were of the one- 
shot case design. 

According to Denzin (1970), the one shot approach 
makes no attempt to approximate the virtues of experimental 
design: no control, no control group, no time dimension 
Rather, it simulates an experiment, not through design but, 
through analysis, specifically multivariate analysis. 

This technique creates analytic groupings out of the 
collected data after the fact, groupings that the researcher 
Should have been seeking initially. The underlying 
vulnerability of multivariate analysis is that any purported 
causal relationships are tentative due to the lack of a time 
dimension. Nagel concludes (1961) that the difficulties of 
basing causal inference on multivariate analysis "are 


MOcCorieus. | 


a. Internal Validity 
73% of the studies reviewed here were vulnerable 
to alternative explanations of their findings because they 
could not control for the effects of history and subject 


maturation in their research designs. 845 were similarly 
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vulnerable to rival explanations of their findings because 
they couldn’t control for the effects of selection bias and 
subject mortality in their designs. 

Further, 22% of the studies were vulnerable to an 
interaction effect of selection bias and subject maturation. 
Another 4% of the works had their findings threatened by the 
effects of instrument calibration and re-test sophistication. 
Only four our of the 45 studies controlled all threats to 
internal validity: two true experiments and two posttest-only 
control groups. 91% of the reviewed studies were in a 
Situation of having their findings open to one or more 
alternative explanations other than those reported by the 
researchers. 

If this group of empirical examinations of system 
effectiveness cannot state, with confidence, that their 
findings are caused by the variables claimed by the 
researchers, but might well be caused by effects that were 
not controlled by the designs of the studies, then they lack 
a basis for establishing causal inference. They cannot be 
the foundation upon which a formal theory of information 


system effectiveness can be built. 


b. External Validity 

89% of the studies reviewed here were vulnerable 
to the possibility that their findings cannot be generalized 
to a population beyond their sample subjects because of the 
effect of selection bias interacting with the experimental 
variable. Their vulnerability to this effect is inherent in 
their research designs. 11% faced the same problem due to 
the effect of re-test sophistication interacting with the 


experimental variable. Only 4% of these studies were, by 
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virtue of their research designs, able to control for any of 
the threats to external validity. Out of the 45 studies 
reviewed, all were vulnerable to at least one of the threats 
to external validity. 

While external validity does not directly affect 
the establishment of causal inference, it does raise the 
question of the relevancy of these research efforts if they 
do not reflect any segment of the real world beyond their own 
samples. Of how much value in the construction of a formal 
theory of system effectiveness will there be in studies that 


speak for nothing more than their own research arrangements? 


3. Time Dimension 

Out of the 45 studies reviewed in this thesis, 89% 
have no way to account for time due to the structure of their 
research designs. In the words of Denzin (1968), studies of 
this sort take a "Snap shot" at one point in time and leave 
it to the respondent to weave a tale about events preceding 
the interview or questionnaire. 

Causal inference requires that empirical findings be 
able to determine that a variable X preceded a variable Y in 
time in order to support the explanation that variance in X 
caused subsequent variance in Y. MIS assessments of system 
effectiveness lack this critical requirement for causal 
inference. Thereby they also lack another foundation for 


building a formal theory of system effectiveness. 


4. Data Collection Techniques 
More than two-thirds of the reviewed studies gathered 
data on system effectiveness variables by asking direct 


questions of human respondents, whether by self-completed 
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questionnaire, survey interviews or by phone. Fewer than a 
third used methods other than the direct asking of questions. 

This renders the greatest part of the MIS studies of 
system effectiveness vulnerable to the possibility that the 
data they recorded reflects something other than the data that 
the researchers believe they collected. The direct asking of 
questions is a method that is obtrusive and socially 
demanding. 

Questionnaires and interviews are suspected of 
creating the very data they record rather than recording data 
on events or processes external to the research act. As with 
internal validity, obtrusive methods raise the issue as to 
whether the reported findings of such studies are due to the 


causes claimed by the researchers. 
a. Survey Research Problems 


(1) Penetrating Language of Group Being Studied. 
Reliance on survey methods - whether self-administered 
questionnaires or personal interviews - raises a number of 
problems. First is that the interviewer or the writer of the 
questionnaire must penetrate the language and symbols of the 
group being studied (Denzin, 1970). Otherwise, the interview 
or the questionnaire are carried out in a relationship in 
which the researcher interacts with the respondents as 
Strangers (Denzin, 1968). This is a process” readily 
appreciated by anthropologists undertaking to study an alien 
culture. It is a process ignored by academics pursuing the 
study of a group that seems to be less remote. The assumption 
by MIS academics that the languages and meanings used by their 


Subjects are already known is a fatal conceit. 
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(2) Differential Status of Respondents. A second 
problem - for which there was no evident concern in the 
studies reviewed here - is that different individuals within 
a group occupy different social positions in a group itself. 
Consequently, their perspectives on the group and its success 
or failures will vary by position. The apparent operating 
assumption in these studies was that members of a group are 
essentially interchangeable parts. Each represents the group 
through undifferentiated expressions. This has the effect of 
yielding responses that are actually quite different from what 


the researchers believe they are gathering. 


(3) Obtrusive Nature of Survey in Everyday Life. 
A third problem concerns the interaction between the 
questionnaire or the interviewer and the respondent. The 
operating assumption in the reviewed studies seems to be that 
the questionnaire or the interview is neutral, eliciting a 
Straightforward recitation of facts or of opinions not 
influenced by the question-asking act itself. 
In actuality, the opposite is the case. 
Respondents are anything but straightforward with question 
askers. This involves socioeconomic, cultural, ethnic or 
Organizational status differences between researchers and 
subjects. Male and female differences is a common source of 
constrained reporting by respondents (Denzin, 1970). So are 
racial differences. These act to create responses out of the 
interaction itself rather than collect data on a phenomenon 


outside the interaction such as, say, systems effectiveness. 
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(4) Social Definition of the Research Process. A 
fourth problem is that the researchers, the topics, the 
respondents, the questions, the answers, the setting, and even 
the questionnaire form are objects within a research 
interaction. Their meaning is not objectively defined outside 
Ge, Or prior to, the research act. Rather, the social 
definition of these objects takes place as a negotiation 
process within the interaction. This is a process about which 
the researchers in this review showed no awareness. 

Again, the data collected may not provide an 
accurate basis for the findings claimed by the researchers. 
As they are described by Reciting (1966), “interviews and 
questionnaires intrude as a foreign element into the social 
setting they would describe, they create as well as measure 
attitudes." Research methods, as Denzin (1968) argues, are 


mete sterile a theoretical tools." 


(5) Role taking in Survey Responses. me fieeh 
problem concerns the credibility of survey responses. As was 
discussed in Chapter IV, people take roles with respect to 
questions. They are vulnerable to Guinea-Pig effects and to 
taking on roles such as antagonist, sympathizer, helper, test- 
taker, informant, protector, participant in a scientific 
endeavor or spokesperson for a group (answering not for 
themselves personally but for an imagined collectivity). 

S These roles put a respondent ina frame of mind 
other than that of a straightforward provider of research 
questions. The latter is probably the very role not taken up 
by respondents. Yet, straightforward provider of answers is 
the only role believed to be presented by survey researchers. 


The result is that the data collected by surveys means are 
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something other than what the researchers believe them to be. 
Researchers analyze these data and report them without any 
suggestion that they are affected by the very process of 
collecting the data. 


(6) Social Desirability Effect. A sixth problem is 
social desirability. This takes place within the interaction 
between a respondent and an interviewer or a questionnaire 
(where the responses are not anonymous or not believed to be 
truly anonymous). In North America, and possibly Western 
Europe, a respondent gives answers that would be socially 
acceptable by a standard known to the respondent (usually 
middle class norms if they are of the middle or lower 
classes). 

In the United States, respondents usually wish 
to present themselves as happy, mentally healthy, educated and 
prosperous. While interview schedules and questionnaires can 
be constructed to detect and discount socially desirable 
responses, there was no indication that this problem was dealt 


with in any of the reviewed studies. 


(7) Fallacy of Reductionism. A seventh problem 
with the survey method is that it necessarily gathers data 
from individuals. Missing are data about social aggregates 
such as organizations, groups, teams or systems. In that 
these 45 reviewed studies purport in some way to examine the 
effects of information systems, it would seem appropriate that 
the data collection would focus on systems per Se. 

What these researchers, did instead, was to 
focus on individuals, thereby andulging the “fatlacy mae. 
reductionism: that a functioning whole is the same entity as 


the sum of its component units. This 2S, in the view@oet 
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Wertheimer (1950), a crucial problem in social research in 
which functional wholes have not been defined as systems the 
behavior of which is not determined by that of their 
individual elements. 

A metaphor suggested by Mill (1879) says that, 
"the different actions of a chemical compound will never be 
found to be the sums of actions of its separate parts." 
According to Nagel (1961) functional wholes are systems which 
cannot be built up out of elements by combining the latter. 
A functional whole cannot be properly analyzed from an 
additive point of view. A whole cannot be inferred from 
properties displayed by its constituents in isolation from the 
whole. 

Coleman (1958) complained that no matter how 
numerous the correlations, how complex the analysis, survey 
studies focus on individuals. They tend to be, in his phrase, 
no more than “aggregate psychology." What these studies miss 
are the relationships between individuals. They show nothing 


about the interactions between people. 


(8) Inability to Measure Behavior. An, Pevgiren 
problem with the survey method is that it is inherently unable 
to measure behavior. re can only record attitudes. As 
discussed in the previous chapter, a survey that asks 
questions about behavior can only record a person’s attitudes 
about the questioned behavior; a survey cannot show the actual 
behavior. This is because, as Denzin points out (1968), the 
emphasis on “respondent reconstruction" of past events, the 
survey is destined to gather data only on attitudes and not 


behavior. 
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Coleman (1969) described this as a shortcut 
taken by most researchers. The study of behavior requires 
observation (watching, participating or experiencing) or 
recording of behavior (archival records, audio tapes, video 
tapes, films). This requires time. It requires equipment. 
It requires digging through records. It requires theoretical 
sampling (Denzin, 1970) of the places where the behavior will 
take place. The convenient assumption behind attitudinal 
research is that there is a connection between attitudes and 
behavior; to wit, attitudes cause behaviors. 

While there is evidence that behaviors lead to 
the development of attitudes, showing that the causality runs 
from attitude to behavior has been problematic (Denzin, 1968). 
On the contrary, evidence has accumulated showing that 
accurate prediction of behavior from measures of attitudes is 
not possible with the methods now employed (Tittle and Hill, 
Poy): ’ 

Studying behavior takes a greater commitment to 
scientific endeavor than is typically required for a one-shot 
design, mail-out questionnaire. Therefore, what is observed 
in these 45 studies, and throughout the general practice of 
research in management topics, is a gravitation to expediency: 
the convenience, the speed, the low cost of the survey method. 

In summary, the survey fails to meet even the 
minimal assumptions for an adequate methodology (Denzin, 
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b. Questionnaire Problems 


(1) Fixity of Questions. in Bddiziem to “Che 
preceding problems of surveys generally that afflict 
questionnaires as well as interviews, questionnaire schedules 
have additional problems peculiar to their usage. The 


reviewed studies that employed questionnaires reported that 


they were of the standardized schedule type. That is, 
respondents were presented with a fixed set of questions and 
a preconceived, precoded, finite set of response 
possibilities. 


The result of this rigid approach was to 
generate numbers, through the questionnaire process, about 
what was on a researchers mind when the questions were 
written. It is a process that taps little of what is on 
respondents’ minds. The likelihood is that the response to 
the indifference of questionnaire fixity is simply fabrication 


(Denzin, 1970). 


(2) Construction of Scales. A second problem that 
is endemic to the questionnaire is the matter of constructing 
scales. The assumption behind the construction of scales on 
attitudinal questionnaires, such as Likert-type instruments, 
is, as Denzin explains it (1970), "that an underlying attitude 
variable is being measured. The scores for any respondent are 
assumed to represent his position on the latent dimension and 
the items are assumed to reflect that dimension." 

Whether the assumption is upheld empirically is 
problematic for all questionnaires using scaled responses for 
MiGMelmloncs Orvdttitudes or opinivens. The difficulty is that 
no single indicator can cover all dimensions of a variable 


(Denzin 1970). A single indicator has only a probability 
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relation to what it is that a researcher actually seeks to 


know. 


(3) Mail Out - Mail Back Problems. A problem that 
plagues the mail out-mail back self-completed questionnaire 
is that nothing is known about the characteristics of the 
people who did not return a questionnaire. This leaves this 
technique vulnerable to the internal validity problem of bias 
in the selection of respondents. A face-to-face encounter 
between question asker and respondent can, and often does, 
involve a "break off" by a respondent. That is, a respondent 
suddenly declines to participate. 

The question asker in these situations at least 
has the possibility of gathering some indications of a 
refuser’s age, sex, race or other obvious characteristics. 
Not so the researcher using mail back responses. The chief 
virtue of the mail out-mail back technique is that is cheap 
and easy. It is not a means for gathering data that will 


Contribute tosthe building scot oumal Cheon. 


5. Definitions of System Effectiveness 

The definitions of system effectiveness gleaned from 
these 45 studies tell a great deal about what is taking place 
in the endeavors of academic MIS to measure the performance 
of systems. The field has polarized into two schools of 
thought: cost-benefit analysis and user attitudes. 

Actually, it is less a case of polarization than it is 
of the aggressive user-attitude camp driving cost-benefit into 
retreat. The user-attitude enthusiasts view do not care if 
their definition of system effectiveness as a resultant of 


scientific endeavor. For them it is a matter of ideology. 
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The user-attitudinalists characterize the use cost- 
benefit analysis as discredited, traditional and infeasible. 
They view user attitudes as modern, feasible and appropriate. 
Their argument, to recall the discussion in the previous 
chapter, is that while data can be gathered on costs, nothing 
can be gathered about benefits. Benefits of information 
systems are so lacking in tangibility as to be rendered 
useless as a way to analyze effectiveness. 

Cost-benefit analysis wasS appropriate for massive, 
batch-processed transactions on mainframe systems. They are 
part these measures of the cost-reduction efficiency 
mentality. What value can have, the attitudinalists ask, when 
we are examining the effectiveness of, Say, a microcomputer- 
based decision support system? 

In the latter case, the emphasis is not’ upon 
efficiency but upon effectiveness. It is not upon quantity 
and cost but upon quality. Therefore, the very advancement 
of hardware technology and the evolution of software systems 
have rendered cost-benefit analysis as deficient and 
inappropriate. 

The only remaining issue, in the minds of the 
attitudinalists, is which Date lecular attitudinal 
questionnaire, and which specific question items, are best 
suited to gauge user satisfaction. That, indeed, is the focus 
of thetrr research endeavors. Not the construction of a formal 
theory of system effectiveness but the fine-tuning of the 
ultimate questionnaire. 

Left out of this approach are any indicators of the 
actual productivity that arises from a system where efficiency 
is the issue or the actual effectiveness of a new system where 


that is the issue. Missing are measures of actual 
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productivity as distinct from users’ Opinions about 
DPmeductivity. 

As the discussion of this issue in the last chapter 
indicated, there is good reason to believe that the conceits 
of the attitudinalists about the superiority of their 
formulation of system success is, in reality. A fig leave 
covering their methodological nakedness. As graduate 
students, one suspects, they invested in learning the 
prevailing but deficient survey method employing standardized 
Likert-scaled questionnaires in mail out techniques operating 
under one-shot case designs. 

The opportunity cost of having learned that approach 
has been an ignorance of other approaches. The 
attitudinalists are not only ignorant of alternative 
approaches, they are ignorant of the problems and deficiencies 
of their own methods. ; 

These suspicions about, and characterizations, of the 
user-attitudinalists would be no more than material for 
empirical testing and academic contention were it not for 
their own aggression. Not content to demonstrate that their 
way might be better, they attack alternative concepts of the 
effectiveness Cf information systems. They force 
practitioners of cost-benefit analysis into a defensive 
posture. 

At the very least, they make difficult an attempt to 
reconcile different views of effectiveness. If the concept 
of efficiency cannot be entered into the dominant dialogue 
about system performance, it cannot be compared or combined 


with user attitudes. 
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B.INTELLECTUAL FOUNDATIONS OF MIS 


1. Failure to Establish Causal Inference 
The choice of research designs, the vulnerability to 
threats to internal validity, the lack of accounting for a 
time dimension and the choice of data collection methods - 
considered separately and taken together - in the MIS 
empirical assessments of information system effectiveness 


provide no grounds for establishing causal inference. 


2. Inability to Build Formal Theory 

The inability to establish causal inference in MIS 
Studies of systems removes a foundation for building a formal 
theory of system effectiveness. By derivation, it further 
removes one basis for constructing a formal theory of 
information systems. Not to construct theory is to miss the 
point of academic endeavor. In the words of Nagel (1961), 
"the paramount aim of theoretical social science is to 
establish general laws which can serve as instruments for 


systematic explanation and dependable prediction." 


3. A Discipline without Theory 

MIS is an academic discipline that cannot construct 
formal theory about its essential concerns. Its has put 
itself in the position of being unable to adequately describe, 
explain or predict the very matters that justify its 
existence. It has traded the pursuit of explanation and 
prediction for the pursuit of career security. Publication 
1s pursued as a means to tenure rather than as a means to 
Supporting MIS practitioners. A gulf will widen between the 
academic world of MIS and the practitioner world of MIS. 


Thereby, does academic MIS flirt with irrelevancy. 
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VI. RECOMMENDATIONS 


A. FOR MANAGERS AND COMMANDERS 


1. The Cost of What Is Not Known About Information 
Systems 
The United States Navy has hundreds of thousands of 
desk-top microcomputers. The very number provokes a concern 
as to how efficiently or effectively they are being used, a 
concern that emanates from high levels in the Navy. Read 
Admiral Paul Tobin, the U.S. Navy’s chief of data automation 
systems, wondered aloud recently how many of the Navy’s 
desktop machines were being used. He was quoted in Federal 
Computer Week as saying, "Let’s make sure we’re using our 
tools. In some way the system has gotten ahead of the users. 
[We] have to figure out how to do more [with the basic 


systems]" (Brewin, 1989). 


Senior military officials, much like senior corporate 
executives, encounter difficulties in dealing with the world 
of information technology. Senior officials often have little 
grasp of the technology of computers. They know that 
information systems can be used to strategic advantage but 
they are at a loss to transform their wishes into successful 
implementations. Therefore some officials delay information 
technology decisions. Others avoid them altogether. Still 
others take the opposite tack and blindly provide a blank 
check for the purchase of information systems, especially 


microcomputer-based systems, without a clear expectation of 
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Gye ctCeturn theyy expece, themeresults they wamt.or how to 
measure the effect of what they getting. 

Admiral Tobin has summarized what most observers of 
the approach by the U.S. military to the new information 
technology already know: money has been thrown at hardware and 
software in the past decade without a sense of what was wanted 
or of what was achieved. That 1s not the same thing as saying 
that the U.S. military has received no return for its 
expenditures. Only that (1) it is unknown what was achieved 
and that (2) consequent achievements will have been the result 
of good fortune and the dedication of field level activity 
system-enthusiasts rather than as the result of planning. 

A good deal of our alleged low productivity and low 
payoff from information systems, especially office automation, 
is due in large part precisely to the generous but 
unscrutinized funding for microcomputer-based information 
technology. Millions have been spent on computers, 
peripherals, supplies, environments and training without the 
hard review that would be given to a turret lathe costing only 
thousands. A lack of scrutiny before the office automation 
funding frenzy has generated more managerial remorse than 
office productivity. In hindsight, managers and commanders 
are aSking, "Where is the computer payoff?" 

This discussion assumes, of course, that senior 
officials truly seek to measure system performance to find 
their payoff. It would be understandable if some officials 
did not want to discover the truth about the productivity of 
office automation. After all, if you have lavished funds on 
desk top computing or information centers, the last thing you 
want to do well or accurately is a study of exactly how much 


productivity was or was not thereby enhanced. 
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2. Becoming Informed Consumers of Information Technology 

In the world of MIS, there exist a division of labor 
between practicing managers and MIS academics. Managers need 
academic MIS to develop scientifically credible ways to 
describe, explain and predict the working of MIS. Managers 
can’t do. It is the job of academic MIS. However, it won’t 
get done if managers indulge the inadequate research methods 
of academic MIS. 

Managers and commanders must re-acquaint themselves 
with the requirements for credible research. They must be 
science literate. They must become informed consumers of 
academic MIS studies of such matters as system effectiveness. 
They must be able to know when reported research results are 
scientifically sound. They must beware of the easy dismissal 
of cost benefit analysis and economic analysis by MIS 
researchers from the user attitude school of effectiveness. 
Managers must be clear about the distinction between an 
effective system and people’s perceptions of an effective 
system. 

Specifically, managers, as consumers of research must 
beware of studies that purport to assess system effectiveness 
by using attitude questionnaires to gauge system outputs as 
well as system inputs. Attitude questionnaires are 
consistent; they exhibit high correlation coefficients within 
their own domains. The trouble is in relating attitudinal 
scores to independent, external grounding in behavior. 

Where MIS researchers deny that data can be gathered 
on system benefits (too intangible they say), it is not only 
possible to gather such data, it is relatively easy. 

A further burden that falls to managerial consumers 


of MIS research is to grasp the place of theory. Managers 
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must not let their own action-oriented, practical mindsets 
mislead them into an impatience with the critical need for 
formal theories in MIS. Managers and commanders must ask 
whether the research results they read are part of an effort 
to build theories that can describe, explain and predict MIS 
system effectiveness. If not, what are the researchers trying 
to do? Are they simply doing studies to gain academic tenure? 

If managers sense that academic MIS, as represented 
by research reports in academic MIS publications, have no 
relevance to their work, they must make that clear to the MIS 
academics. Unfortunately, managerial MIS does not pay for 
academic MIS directly. The income of academic MIS is not 
dependent upon the pleasure of practicing MIS. This structure 
of third party financing of research allows MIS academics to 
be distracted from the construction of theory into the 
expedient pursuit of careers. 

However, MIS practitioners do provide academic MIS 


with its legitimation, its reason for existing. Legitimation 


womimr a Strong control in the short run. mn the long run, 
without acceptance by practicing MIS, academic MIS is 
irrelevant. 


This situation is different in the interface between 
MIS managers in the defense community and academic MIS 
researchers attached to defense educational institutions. 
There, ~-the practitioners provide not only legitimation but 
also research funding. 

MIS academics in the defense community are held 
accountable for their research results. While this control 
possibility is not perfectly exercised, its potential is large 


for disciplining academic MIS research. 
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B. FOR THE STUDY OF SYSTEM EFFECTIVENESS 
1. Research Designs 


a. Time Dimension 

Until MIS research designs for studying system 
effectiveness account for the time dimension, they will lack 
any basis for causal inference (Vitalari, 1985). All MIS 
research designs must gather data at more than a single point 
in time. The portion of the MIS studies that have no concern 
with time in system assessment (now 89%) must be reduced below 
25% in the next decade. Otherwise, MIS research may fail to 
establish scientific credibility. MIS researchers must 
abandon cross-sectional analysis designs in favor of 


longitudinal designs. 


b. Randomized Comparison Groups 

As Nagel (1961) noted, it is possible to advance 
scientifically without meeting the full requirements of a true 
experiment, yet some form of controlled investigation "appears 
to be indispensable." To that end, MIS researchers must not 
only employ comparison groups in research designs (which 27% 
of the reviewed studies did) but use comparison groups with 
randomized assignment of respondents. 

Only by that means can MIS research control for the 
threats to internal validity arising from history, subject 
maturation, re-test sophistication, instrument calibration, 
statistical regression, selection bias, respondent mortality 
and the interaction of selection bias and subject maturation. 
Only then can MIS research build a confidence that its 
findings actually reflect what MIS researchers intend instead 
of being open to rival explanations by effects which were not 


taken account of in the research designs. 
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Cc. Quasi-Experimental Designs 

While a great deal can be learned from laboratory 
experiments, the rich and fruitful MIS research of the future 
will be done in field settings. Most of it already is (96% 
of the reviewed studies were done in field settings). The 
difficulty is that the experimental virtues of repeated 
measures with randomized controls are not now being 
approximated. 

The route to gaining the richness of field setting 
research with the rigor of experimental controls is quasi- 
experimental research designs (Campbell and Stanley, 1966). 
MIS researchers must first declare the experimental variable 
they are examining. They must then gather data before the 
application of an X variable and again after. They should 
make longitudinal (multiple observations), especially in the 
posttest phase. This is not a difficult requirement. It can 
be accomplished with survey methods as well as with 
unobtrusive methods. The beauty of this approach is that 
information systems inherently produce data on key variables 
that a researcher into system effectiveness will want to 
measure. 

To compare organizational functioning in a manual 
work mode with that in a post-implementation mode, researchers 
must commit to digging through accounting? and other 
organizational, hard-copy archives. Ing the pest- 
implementation phase, they can then indulge the relative 
luxury of having the system being studied provide the data. 

For an example of this technique see the one-group 
pretest/posttest study of a standardized contracting and 


purchasing system by Barclift and Linson (1988). 
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2. Triangulation 

Triangulation is amply illustrated by the proverb of 
the blind men reporting on the characteristics of an elephant. 
Each man reported different characteristics, depending on 
whether he was making a tactile examination of the trunk, 
tusks, legs or tail. The situation is much the same in MIS 
research. Each researcher, employing a single method of data 
collection, a non-experimental design and a single theoretical 
position on the nature of system effectiveness is little 
better than a blind man. Only conceits and psychic 
investments in current methods keep MIS researchers from 
appreciating the limits of their empirical insights. 

When the proverbial blind men can compare, contrast 
and locate their findings, they open the potential to 
describing the illusive reality of the elephant. Once MIS 
researchers use multiple methods of data collection, multiple 
observation points in time, multiple comparison groups and 
multiple forms of data then they, too, will be open to the 
rounded, realistic description of the demonstrated (rather 


than the assumed) effectiveness of information systems. 
a. Triangulation of Data 


(1) Triangulation of Time. Of the reviewed 
studies, specification of time was inadequate. The most 
specific mention was the year in which data were collected. 
Beyond that, no attention was given to the timing of data 
eoilectione Triangulation of time would require data 
collection with these variations: different times of year, 
different times within a month (a great deal of information 
system activity has a monthly cycle), different days of a week 


and different times during a day (this is a potent but 
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neglected avenue of research into, for example, the way a 
night shift crew uses a hospital information system without 
the support of daytime MIS staff). 

Not all of these variations may be relevant to 
any particular examination of an information system. However, 
the likelihood is that one of them will profoundly affect the 


data gathered from any one type of information system. 


(2) Triangulation of Place. All human activity has 
a Spatial designation. Many of the reviewed studies noted 
the organizational location of their data collection that 
often implied a geographical variation. Yet, spatial 
differences were not introduced consciously as a variable. 
Space differences are attended to cross-cultural comparisons, 
but cross-cultural comparisons have not been a part of system 
effectiveness inquiries. 

The temptation in MIS research is to assume 
that place is irrelevant without further consideration. The 
conceit is that influence of information technology overcomes 
differences in time, culture or place. But does it? Do 


studies exist that establish that spatial location can be 


excluded as a critical independent variable? No. Instead, 
place is conveniently assumed = away: MiLormation aS 
information. 


Yet, information is an object in our lives 
subject to the same, situated negotiations among interacting 
people as any other object (Boland, 1985). The influence of 
Spatial location is an empirical question; until its effect 


is deliberately examined, it cannot be assumed away. 
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(3) Triangulation of Level of Data. This thesis 
has already exclaimed upon the need to move MIS studies on 
system effectiveness away from data collected from individuals 
to data collected about systems and organizations. A 
triangulation of data would require that any study of systems 
performance include data from three levels: (a) individuals 
(which MIS studies now gather in abundance and to the 
exclusion of other levels of data), (b) systems (departments, 
teams, shifts, and true systems of people who are connected 
in relationships that are built around producing an output 
through the mediation of hardware and software of their 


information technology no matter what their formal 


organizational identities) and (c) interactions (the 
encounters, negotiations, defianatcions: avoidances, 
competitions, cooperations, and coordinations between 


individuals or groups but not characteristics of the people 


= 


or the groups per Se). 


b. Triangulation of Theories 

To speak of triangulating theoretical explanations 
of system effectiveness is premature in a discipline that has 
not developed a formal theory nor is presently able to do so. 
Yet, we have emerging, competing schools of thought as to how 
the concept of system effectiveness should be defined. Any 
study of system effectiveness ought, therefore, to approach 
an empirical assessment with more than one concept of 
effectiveness. 

A system’s performance must be studied in terms of 
user attitudes. At the same time it must be studied in terms 
of costs and benefits. Further, it must be studied in terms 


of the economic value of information. 
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And, finally (without exhausting the MIS inventory 
of concepts of system effectiveness), it must be studied in 


terms of organizational effectiveness. 


c. Triangulation of Methods 

survey researchers already employ a variant of 
methodological triangulation - within-method triangulation 
wenzin, 1970). They use various scales and multiple 
standardized schedules to tap attitudes they are examining. 
The limit here is that this takes place within a single 
method. This leads researchers to congratulate themselves on 
the breadth of their methodological labors while deluding 
themselves that variations on the same method produce distinct 
viewpoints. According to Webb (1966), each method is 
imprisoned within its own biases and distortions. Combining 
within-method variations is a first step but it cannot be the 
only step. 

What is required of MIS research is the combination 
of methods to study the same system or the same phenomenon. 
Thereby wili the strengths of one method compensate for the 
weaknesses of another. The effectiveness of a information 
system implementation can easily be assessed using unobtrusive 
measures as well as, say, attitudinal questionnaires as 
Barclift and Linson (1988) did. 

The greatest benefit of between-method 
triangulation is countering rival explanations for reported 
findings that arise from reactive arrangements. The reactive 
effects of questionnaires and interviews on data were 
discussed in the previous two chapters. In an ideal strategy, 
MIS researchers will combine survey questionnaires, personal 


interviews, archival data, system-generated measures, 
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participant observations and phenomenological techniques 
(Webb, 1966; Denzin, 1970; Boland, 1985). 


3. Long Term Field Studies 

Even with a recommendation for accounting for a time 
dimension and for triangulating for time as a data variable, 
the preceding discussion has been framed in the assumption of 
a relatively short term research project measured from months 
to as long as two years. Covering the implementation of an 
information system, from system proposal to final, post- 
training full implementation could easily cover two to three 
years. But MIS research must commit to going beyond that. 

Once the introduction of a system has been assessed, 
MIS researchers must continue with follow-up studies through 
the life cycle of a system. Considering that the useful life 
cycle of some information systems is quite short (most are 
shorter than a decade and some scarcely last three years 
before succumbing to obsolescence), this is not a burden. 
Once entry has been made to an organization to study its 
information systems, the greatest hurdle is past. Unless MIS 
researchers have a propensity to be wearisome to their hosts 
and wear out their welcomes, they must extend their research 
relationships for a long view. 

The payoff form this is that different things happen 
to information system in the long term than are ever imagined 
in the short term. The most potent revelations in social 
research have come from the longest studies: communities, 
organizations, cohorts of people over a lifetime. The same 
will be true with MIS topics. In their examination of the 
introduction of a productivity-enhancing system that cut 


contact-processing time in a U.S. Army installation, @lmmsan 
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and Barclift (1988) suggested that follow-on research was 
needed to see if the short term efficiency, in itself, would 
provoked user activities to make greater demands on the system 


in the years ahead. 


4. Unobtrusive Measures 

A theme that has been exercised in the fourth and 
fifth chapters of this thesis, and echoed earlier in this 
chapter, is the distorting, biasing effects of the survey 
method. This leads to a recommendation for MIS research to 
embrace the methodological point of Webb at al (1966), for 
the use of data collection techniques that are not reactive 
- that is, that do not lead to role-playing, fabrication, 
social desireable response and other behaviors that arise in 
the research interaction. 

This recommendation is for the use of nonreactive data 
sources, specifically archival data sources. aes 
recommendation 1s particularly appropriate to MIS research 
because the production of information is the point of 
information systems. Some of that information reflects the 
performance of the system itself. This does not mean the 
technological monitoring of hardware system in terms of 
processor clock speed or input/output efficiency. Rather, it 
means the use of organizational accounting data on costs, 
production and outputs. 

~“An argument will arise from the user-attitudinalists 
that the archival data produced by information systems will 
miss the intangible benefits of an information system such as 
higher morale. Once again this reflects an investment ina 


Single method rather than the reality of available data. 
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One approach to measuring morale is to use attitude 
questionnaires that purport to assess it. But only attitudes 
will be assessed. Do the attitudes inform the behavior of 
people? Does it direct what they actually do? As an example 
of an alternative measure of morale, examine organization 
records for the classic workplace-safety-valve for people who 
are unhappy with their jobs: sick leave (Barclift and Linson, 
1966). When people are truly are fed up with their work, 
especially people below the managerial or professional levels, 
they avoid the workplace by staying home and calling in sick. 
Or, they drive themselves into actual symptoms of illness 
produced from anxiety and job stress. Why then must an 
attitude questionnaire by employed exclusively to assess 
morale when a powerful, triangulating behavioral measure is 


sO available? 
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questionnaire actually captures the concept it purports 
to measure. 


The first role of the UIS measure is to detect the 
presence of a problem with user satisfaction in an 
organization. This would be achieved by administering a 
UIS survey and examining the total satisfaction scores. 
It 1s critical that the interpretation of UIS results be 
treated cautiously and contextually. 


The UIS measure provides a way to obtain information about 
overall user satisfaction with information services as 
well as more focused information on specific satisfactions 
in the three areas of EDP staff and services, information 
product, and user knowledge and involvement. 


The real value of a UIS meaSures lies in its ability to 
discriminate among a large number of possible problem 
areas. The situation and the purpose of the study should 
guide whether the manager/researcher choose the long form, 
the short form, or a simple one-question UIS measure. 
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Beuwen, ©. Uso. , 
A Descriptive Model of Success for Computer-Based 
Information Systems 


Information and Management, vol.1, no.2, pp.63-67. 
July 1984. 


Post implementation evaluation of computer-based 
information systems is of vital importance for improving 
the operation of current systems and in the design of new 


applications. 

Three major activities in which this plays an important 

Dalat: 

ie. The acquisition of equipment. 

a. In improving the operation of production systems. 

oF, In the design of new applications; this is extremely 
important, both to the computer staff and users of the 
system. 


A cost-benefit study is one of the best ways to measure 
it. Obviously, the benefits from a computer-based system 
Should exceed the cost of developing and running it. 
Unfortunately, assessing the benefits has been difficult, 
particularly for systems supporting sophisticated 
decisions; i1.e., beyond the routine processing of 
transactions. 


Cost-benefit is best but too difficult - so use user 
satisfaction. Use and satisfaction are relative, that is 
they are measure on a continuum, not a binary scale. 


Colton, Kent W. 
Computer and Police: Pattern of Success and Failure. 


Sloan Management Review, vol.14, no.2, pp.75-98. 
Winter 1972-73. 


Inventory of problems, not a measure of success. No 
metric of effectiveness, only impressions. 


There are indications that computer use may lead to 
centralization of power. Of those interviewed during the 
on site visits, 96% felt that computer use would lead to 
better information and grater control of the police 
department by those at the middle and top management 
level. 


It should be remembered that the field of computer 
Beemmolvodgy 1S still in its infancy. Perfection should not 
be expected instantly in an area so young and rapidly 
changing. On the other hand, there is a certain mystique, 
as well as commercial force, surrounding the application 
of computers. This has led to high expectations and in 
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some respects to oversell. As a consequence, there is a 
need for a continuing assessment of the benefits and 
impact of computer use. 


Cron, William L. and Marion G. Sobol 
The Relationship Between Computerization and Performance: 
A Strategy for Maximizing the Economic Benefits of 
Computerization 


Information and Management, vol.6, no.3, pp.171-181. 
June 1983. 


Surprisingly the immense growth in business usage of 
computer facilities and software in the past decade has 
not been accompanied by much analysis of the influence of 
computer ownership and compute usage on the sales, costs, 
and profitability of the firm. Part of the problem is one 
of cause and effect: whether higher computer usage leads 
to more profit or whether higher levels of profit lead to 
more computer usage. A second difficulty is that of 
estimating the sales or profits if computer systems had 
not been introduced. ; 


They never discussed how performance should be measured 
or related to these variables. 


Other attempts at evaluating performance of information 
systems have relied on management opinions, based on user 
surveys. Thus Neumann and Segev state "a universal 
measure like net effect on earning cannot be applied yet". 
In summary, the literature shows that relatively few 
studies of these effects have been made. 


Result of this study indicate that computerization is 
positively related to the overall performance of medium 
and large size firms. Small firms should be cautious in 
estimating the potential for significant economies of 
scale from computerization, but should use information to 
facilitate management control and decision-making. 


Decker, Jane E. and John P. Plumlee 
Microcomputers, Manpower and Performance in a Public 
UEatai cy 
Public Productivity Review, vol.9, no.2-3, pp.213-225. 
Summer/Fall 1985. 


Rarely, if ever, can managers simply deploy microcomputers 


into the organization and receive a clear-cut productivity 
bonus. 
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The Performance Improvement Division sought and acquired 
microcomputers not to create a system but to enhance its 
own productivity within that system. 


The JEA experience demonstrates dramatically that the new 
technology of microcomputers provides real opportunity for 
the enhancement of public sector productivity. 


Durand, Douglas E., Rex 0. Bennet and Samuel Betty 
Siat Does Information technology “Do™ to Business 
Communications?: Two Empirical Studies 


information _ and Management, vol.13, no.3, pp.111-117. 
@@tober 1987. 


MeeeesuUseprisingly little empirical research has been 
@en@ucted to determine the effect of information 
technology on business communications. 


Productivity can be divided into two segments: efficiency 
and effectiveness. Both studies suggest that end-user 
word processing communications result in different output 
than those produced by conventional means. 
Organizationally, productivity will decline unless the 
communication is more effective. Does the write’s 
communication style effect the manager’s confidence in 
decisions? Thus, important questions remain regarding the 
effectiveness of communications technology. 


moaelrman, Franz. 
Managers, Computer System, and Productivity 


M_eemovarterliy, vol.5, nowS, pp.1-19. 
September 1981. 


One measure of system’s effectiveness is the degree of its 
Organizational impact. 


The best load to more efficient office operations is 
through more effective support the managers. 


Bach dollar invested at the top of the pyramid in 


effective system technoloay can be expected to replace $X 
@eedollar) in white collar labor cost. 
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Edstrom aeanders =. 
User Influence and the Success of MIS Projects: A 
Contingency Approach. 
Human Relations, velwsu, Mo.7, pp-567-007. 
1977. 


There are several reasons why one has to resort perceptual 

data: 

pes Generally one can estimate the cost for developing 
a system but not the benefits, mainly because a 
large number of these are intangible. 


a Managers do not seem to keep track of these data and 
hence they are not available for research purposes. 
cy Even if we should know the complete cost-benefit 


picture, it still would be impossible to judge the 

relative success with respect to alternative 

solutions to the problem in any objective sense. 
The information needs of the manager are thus crucial for 
the design of the system, and his satisfaction with the 
outcome is certainly a very important criterion of the 
success of the system. 


The perceived success of an information system is, 
however, not only possible criterion of success and it 
seems to be useful to investigate alternative aspects of 
the success. 


The phase of the development process is a further 
contingent factor and that it is important to distinguish 
between types of actors in the development process. 


Englander, Valerie and Fred Englander 
Cost-Effectiveness of Computer-Based Welfare Detection in 
New Jersey 


Public Productivity Review, vol{9,"no.2-Ss) ppe2zs i202 
Summer/Fall 1988. 


The New Jersey program... entails a quarterly computer 
match of the social security numbers of welfare 
recipients... against the social security numbers of those 


On public and private payrolls. 


Given that the fraud detection system was implemented 
Simultaneously with major national welfare reform 
legislation, the New Jersey program was assessed by 
examining patterns in N.J. caseloads and average grant 
levels relative to the national trends. This methods 
generated an estimated savings of $45.6 million, 
substantially greater than the $6.2 million cost of 
operating the program. This is an encouraging result to 
those policy makers who have recently sought to extend 
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computer-based approaches to detect and prevent fraud in 
other Federal programs. 


Epstein, Barry J. and William R. King 
An Experimental Study of the Value of Information 
Onlegqa, vol.10, no.3, pp.249-258. 
w9'8 2 . 


Virtually every controllable aspect of any information 
system, its design and implementation, are affected by how 
information is valued. 

No satisfactory measure has been developed that has 
general applicability and the question of which measure 
should be applied in a given circumstance has not been 
adequately addressed. 


Most measures of the value of information apply to a unit 
that is at a low level of aggregation. Tapeh ces” the 
sort of information value concept that must be used in 
designing an IS, the designer is left either to design 
systems idiosyncratically for individual, or to aggregate 
information values across individual in order to construct 
a single measure that best reflects the organization’s 
information values. 


Franz, Charles R. and Daniel Ruby 
Organizational Context, User Involvement, and The 
usefulness of Information Systems 
beciusaoen Sevences, vol.l?, now’, ppre2o=s55. 
Summer 1986. 


Two factors reflecting an MIS department’s maturity (size 
and age) were found to reduce users’ perceptions of system 
usefulness. 


The role of organizational factors was more complex than 
first imagined. Failure of the second canonical 
correlation to produce significance led to the rejection 
of all our hypotheses about direct associations between 
organizational context and user involvement. 


Suggested to strengthen the positive relationship between 
involvement during design and perceived usefulness. 
Paradoxically, MIS departments with broader scopes seemed 
to weaken this same relationship. 


Fuerst, William L. and Paul H. Cheney 
Concepts, Theory, and Techniques: Factors Affecting The 
Perceived Utilization of Computer-Based DSS in the Oil 
Industry 
Decision Sciences, vol.13, pp.554-569 
1982. 


Perceived usage, for the purpose of this study, was 
defined as the decision makers’ perception of how 
frequently they interacted with the DSS and how important 
the information obtained from the DSS was in assisting 
them to make decision. 


In terms of general use, the factors of user training 
during the implementation process and the accuracy of 
output were found to be important. 

The only factor of importance from the characteristics of 
the decision maker was the number of years of experience 
in the present position, and it was only important for 
specific DSS use. Thus the greater their years of 
experience in their position, the more managers will use 
the DSS for specific, personally initiated reports. 


The important factors in the characteristics of the DSS 
are accuracy and relevancy of output. 

Training and accuracy were found to be important in 
affecting general DSS use; and experience in the present 
position, user training, accuracy of output, and relevancy 
of output were important in affecting specific DSS use. 
Thus, the system’s professional should stress’ those 
factors when designing, building, and implementing a DSS 
in order to maximize the degree to which decision makers 
use the system. 


Gallagher, Charles A., 
Perceptions of the Value of a Management Information 
system. 


Academy of Management Journal, vol.17, no.1, pp.46-55. 
March 1974. 


Difficulty measure benefits.A dollar comparison of costs 
with benefits cannot be made. 


This study included one test of a new methodology. The 
result of a single test cannot establish the validity of 
a methodology. The study results suggest that the 
technique is basically useful in this application, nut 
better scales need to be found. 


The finding of this study that managers in certain upper- 
level managerial positions valued EAB reports more highly 
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than other managers suggest the possibility of a general 
relationship between managerial position and the value of 
information. Such a general relationship, if it exists, 
should vary with the type of information and the type or 
organization structure. 


Gallupe Brent R. and Gerardine DeSanctics 
Computer-Based Support for Group Problem-Finding: An 
Experimental Investigation 


Mis Ouarterly, ppaZ71-296. 
June 1988. 


Two versions of the experimental task, one higher in 
difficulty and the other lower in difficulty, were 
administered to GDSS-supported and nonsupported decision- 
making group, yielding a 2 x 2 factorial design. 


Groups with the higher difficulty task were less satisfied 
than those with the lower difficulty task; however, this 
difference was not’ significant. The anticipated 
interaction effect between GDSS support and_ task 
difficulty was not observed. 


The only meaningful finding for this variable was that 
GDSS support resulted in a significantly lower level of 
Satisfaction with the decision making process. 


As controlled experimental study the findings cannot 
necessarily be generalized to all levels or types of GDSS 
software or to actual organizational use of GDSS 
technology. 


Hcg, P. G., and T. J. Lincoln 
A Cost Effectiveness Study of Production and Engineering 
System within a elon che TY, integrated Manufacturing 
Environment. 


Management Datamatics, vol.5, no.6, pp.262-274. 
1976. 


SESAME was primarily established to evaluate the cost 
effectiveness of the computer systems at BAC-CAD. Any 
cost effectiveness study assumes a comparison of one 
Situation with another. Although the most straightforward 
method of comparing costs and benefits would have been to 
compare the current computer systems with the cost of the 
number of people required to do the same job, it was felt 
that this would be a meaningless exercise. 
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These considerations suggest that the cost and benefits 
of the computer systems are best compared with the 
realistic manual systems that would probably be introduces 
if the computer was not available. 


The basic philosophy of a SESAME study is that information 
for management iS, in our ever more complex environment, 
rapidly becoming one of the most important of company 
resources and that good management demands the most 
effective utilization of the information resource that can 
be professionally obtained. 


Hogue, Jack T. and Hugh J. Watson 
An Examination of Decision-Makers’ Utilization of 
Decision Support System Output 


Information and Management, vol.8, no.4, pp.205-212. 
August 1985. 


Decisions are considered to include three major phases - 
intelligence, design, and choice. In order for a DSS to 
support the intelligent phase, it must provide information 
Gf a deseriptive tatuces 


The primary justification for the development of a DSS is 
that it will be of value to the decision maker. 


Monetary assessment of success is probably not likely 
Since it is rare for a DSS user to formally evaluate the 
impact of the DSS on profits. It may be that the use of 
a DSS constitutes a successful system. This study has 
Suggested some combination of satisfaction, acceptance, 
and frequency of use as potential indicators of success. 


Kapsales, Peter 
Before and After WP: An Office Automation Productivity 


Study 


Journal of System Management, pp.7-9. 
April 1986. 


If I buy $70,000-worth of word processing (WP) equipment, 
my operators must still type the names and addresses. How 
much productivity 1s gained by word processing, when the 
number of key strokes is not reduced? 


Managers must decide what the potential gain in 
productivity is, and if it is large enough to justify the 
cost of the new equipment. It 1S a Gifflcwmlt decmsion 
when there is not a reduction of keystrokes. 
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Is Word Processing Productive? 
WP equipment is justified when it: 


a. Saves the typist from making unnecessary keystrokes. 

Dy. Speeds up and simplifies the document formatting 
activities. 

om Provides significant benefits by generating letters 


which are personally addressed to customers. 
Automating an office almost always provides productivity 
improvements, but careful analysis should be performed to 
make sure the benefits outweigh the costs. The manual 
environment and the impact automation will have on it must 
be carefully evaluated. 


King, William R. and Barry J. Epstein 
Assessing Information Value: An Experimental Study 
Decision Sciences, vol.14, no.1, pp.34-45. 
January 1983. 


Comparing opinions of total system with a linear model of 
opinions about separate attributes of system. 

Big error: Calls a scale internal after explicitly 

directing respondents to treat it as ordinal. 


This study demonstrates the practicality of using a 
multiattribute approach to the evaluation of information 
systems. These evaluations were compared with this 
calculated from a multiattribute linear value model. The 
two evaluations were found to be highly correlated. 


The important notion of considering alternative system 
designs is often not operationally applied because of the 
complexity and cost involved in describing alternative 
systems that are directed toward the same managerial 
objective. The ease with which systems may be described 
in multiattribute terms suggests that the use of this 
approach may facilitate the explicit consideration of 
alternative designs. 


King, William R. and Jaime I. Rodriguez 


MIS Quarterly, pp.43-51l. 
September 1978. 


Most evaluations of information systems are provided only 
in efficiency-oriented terms on a post hoc basis by system 
users. 


Attitudes and value perceptions are an important and often 
neglected aspect of MIS evaluation. 
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The MIS evaluation process is thus a dynamic one in which 
each systematic evaluation effort can lead both to better 
information systems and to improved evaluation 
methodologies. 


Larcker, David F. and Parker V. Lessig 
Perceived Usefulness of Information: A Psychometric 
Examination 
Decision Sciences, vol.11, pp.121-134. 
1980. 


Perceived importance will refer to the quality that causes 
a particular information set to acquire relevance to the 
decision maker. If the information items are a necessary 
input for tasks accomplishment, then this quality will 
tend to increase the perceived usefulness of the set. 


Perceived usableness will refer to the information quality 
that allows a decision maker to utilize the set as an 
input for problem solution. If the information is clearly 
presented and readable or can be easily transformed, then 
this quality will tend to increase perceived usefulness. 


A number of instruments have been employed to "measure" 
the perceived usefulness of information for decision 
making, but these techniques lack any data on the 
instrument reliability and validity. Thus in general 
sense, the previous techniques are inadequate and lead to 
a lack of comparability in research results. 


LiInegolne i. 
Do Computer Systems Really Pay-off? 


Information and Management, vol.11, no.1, pp.25-34. 
August 1986. 


Despite the wide use of cost-benefit forecasting to 
justify proposed system investment, executives remain 
skeptical of the level of benefits actually achieved. 


Such a situation reinforces suspicion in executive’s minds 
that benefits from computer systems are speculative and 
probably marginal. 


Post-audits of established systems, however do not have 
such an obvious basis for comparison and it is essential 
to decide what the comparitor will be before undertaking 
a study. 
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SESAME studies however are concerned with overall cost- 
effectiveness and it is usually possible for system 
boundaries to be defined which are meaningful to 
Management and allow fairly straight-forward cost 
allocation. The objective is to produced a cost profile 
which is accepted by senior executives as realistic and 
comprehensive. 


Increasingly we see a reluctance to invest further until 
concrete evidence is available that existing system 
investments have produced reasonable financial returns. 


User benefits are frequently declared as "intangible" and 
rarely post-audited. 

All too frequently there is a lack of financial discipline 
applied to the system development process: it is often 
unclear who is accountable for benefit achievement. 


EucaemJr., Henry C., 
System Quality, User Reactions and The Use of Information 
Systems. 


Management Informatics, vol.3, no.4, pp.207-212. 
1974. 


The findings of this study suggest that before undertaking 
new systems, the quality of existing systems as rated by 
users should be satisfactory. It may be necessary to 
enhance or redesign existing systems before undertaking 
the development of new ones. 


A user’s past experience is his best predictor of the 
benefits of a new system, and successfully functioning 
systems are prerequisite to obtaining user cooperation in 
system design. 


Though causality has not been demonstrated, a priory 
arguments and the data support the goals of developing 
favorable user attitudes and high quality systems to 
encourage the use of information systems. 
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bUGdseuwts, Henry C., 
Performance and The Use of an Information System 


Manademem= ocience, vol.21, no.8, pp.908-919. 
Aprerl 1975. 


Information can be used both to find and solve problems, 
though most current information systems seem to _ be 
oriented toward problem finding. Possibly problem finding 
data can be provided under exception conditions specified 
by users. 
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One of the most important implications of the model and 
results is that different personal, situational and 
decision style variables appear to affect the use of 
systems. These findings argue for more flexible systems 
to support different user’ needs. 


If attitudes are highly negative, on the other hand, users 
will minimize the contribution of the system and question 
the validity of output reports. Attitudes and perceptions 
of computer systems should be influenced in turn by the 
quality of the system as perceived by users. 


Lucas Jr., Henry C. 
Application and Implementation (Unsuccessful 


Implementation: The Case of A Computer-Based Order Entry 
System). 

Decision Sciences, vol.21, no.8, pp.908-919. 

Apes Aoi: 


Dependable variables have usually focused on some measure 
of the level of model or cdOmputer system usage. Where use 
1s inappropriate, researchers have employed measures of 
Satisfaction and/or favorable user attitudes as indicates 


of successful implementation. 


Lucas Jr, Henry C., 
The Use of Accounting Information System, Action and 


Organizational Performance 


The Accounting Review, pp.735-746. 
October 1975 


In a field study it is difficult to obtain good measures 
for operationalized variables and to control for 
confounding variables. The lack of extremely strong 
results can be partially attributed to errors in measuring 
the variables which were included in the research. 


Laboratory experiments make it possible to control better 
for confounding variables, particularly in measuring 
performance. 139) addition Ee interviews and 
guestionnaires, the use of protocols may be necessary to 
develop an understanding of how information is integrated 
with the decision-making process. 
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Mansour, Ali H. and Hugh J. Watson 
The Determinants of Computer Based Information System 
Performance 


Beademy of Management ®Journal, vol.23, no.3, pp.521-533. 
September 1980. 


The effectiveness of the relationship between system 
personal and others in the organization have an impact on 
system performance. (They looked at from Behavioral 
Variable). 


Mason, Richard O., 
Measuring Information Output: A Communication Systems 
Approach 


Information and Management, vol.1, no.5, pp.219-234. 
October 1978. 


Notes, that measures of behavior require Girect 
observation. The most frequently used measures of change 
in behavior are measures of ability to perform problem 
solving tasks and tests of skills, although most measures 
of change in behavior are unique to each situation and are 
Gar ticuit to identify inwa compex situation. 


Productivity may be measured as a ratio of useful output 
to the total number of executions of a unit operation. 
Our concerns for efficiency require that the output of 
these systems be related to the input resources consumed. 
Only then can managerial judgments be made as to the 
appropriateness of the allocations. 


There is an important class of systems in our society, for 
which output measures are more difficult to define. These 
are systems whose primary output is information. 


Matlin, Gerald. 
What is The Value of Investment in Information System 
Pitomouarterly, vod. 3g no. 3,ppro-s4s 
September 1979. 


Difficulty in finding a value is not an appropriate reason 
for avoiding the subject. 


Are they getting what they paid for in MIS? 


What is the value of the company’s investment in 
Information Systems? 
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Managers intuitively know that the most of the benefits 
of IS investments are intangible, difficult to measure, 
and difficult to relate to profit result. 


If the expenditure for the system investment is approved 
by appropriate levels of management and if other valuation 
methods are not appropriate to identify all benefits, then 
the value of the IS investment is equal to the standard 
of ROI in the company times the cost of the IS investment. 


Miller, Jonathan and Brenda A. Doyle 
Measuring the Effectiveness of Computer-Based Information 
System in the Financial Sector 
MIS Quarterly, vol.1l,) nmOmt eo 124. 
March 1987. 


It was found that performance varies widely between firms 
and that the better performers were those where the level 
of performance matched the perceived importance of the 
particular aspect of IS. 


IS effectiveness here is, at least in part, a function of 
the relationship between perceived importance and 
performance on individual information systems attributes. 


Money, Arthur., David Tromp and Trevor Wegner 
The Quantification of Decision Support Benefits Within the 
Context of Value Analysis 


MIS Quarterly, pp.225-coGr 
June 1988. 


A DSS that is perceived to have value by the potential 
users 1S more likely to be adopted and implemented than 
one for which no Significant value is perceived. 


The technique used, namely conjoint measurement, is 
considered a robust procedure for converting subjective 
judgments about the relative importance of intangible 
benefits into numeric scores for further analysis. 
Traditional cost-benefit approaches to evaluating the 
effectiveness of decision support systems are generally 
regarded as inadequate. More recent approaches account 
for the intangible benefit analysis, but still attempt to 
express benefits in terms of costs. 


Consequently, a more rigorous interpretation of the 
perceived benefits of a DSS, which are a measure of value 
of the proposed DSS, should greatly facilitate the 
decision on whether to proceed with the next stage of 
development or acquisition. 
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Neumann, Seev and Eli Segev 
A Case Study oof User Evaluation oof Information 
Characteristics for Systems Improvement 


Information and Management, vol.2, pp.271-278. 
noo . 


The user of the information and his needs are the key for 
the system design. If the system does not fit the user’s 
needs, it will not be used. Extent of use serves in this 
case as an indication of the success of the system. 


Improvement of an existing system is aimed at providing 
a better response to user needs, that is, increasing the 
use of the system by increasing user satisfaction from 
existing system, it is necessary to first measure his 
current evaluation of the system, find out those aspects 
of the information which receive low evaluation, and then 
improve them. 


The findings of this study indicate that the users at the 
bank had a holistic perception of their information, and 
that content was a dominant characteristic which spilled 
over other characteristics. 


Poppel, Harvey L. 
Who needs the office of the future? 
Harvard Business Review, vol.54, no.5, pp.146-155. 
November-December 1982. 


Almost everyone could benefit from the new technology, 
aeeording to this study of white-collar productivity. 


Despite the allure, however, businesses are spending less 
than a nickel for such technology of every dollar of 
knowledge-worker salaries and internal support costs. 


The fact is that most decision makers are skeptical about 
what managerial work stations, personal computers, video- 
conference rooms, and the other, newer icons of office 
automation can do for their businesses. These executives, 
disenchanted by their pervious exposure to ill-conceived 
forays into management information systems and word 
processing, doubt that knowledge workers will embrace the 
new technology, and they lack confidence that their 
organizations can channel and measure the intended 
benefits. In addition, members of the computer illiterate 
majority are worried about whether, when, and how they 
themselves will deal with these new electronic tools. 
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Raghunatan, TS. and W. R. King 
The Impact of Information Systems Planning on the 
Organization 
Omega, vol.16, nowmzeieep .25- 96. 
September 1987. 


"...Attempts to relate planning to overall business 
performance, measured in terms such as_- return-on- 
investment (ROI), are not fruitful, primarily because even 
such an important activity as strategic business planning 
is only one of many factors that influence ROI." 

These studies, together with the limited ones done in an 
IS planning context, suggest that IS planning may be most 
fruitfully evaluated using a _ surrogate measure of its 
organizational impact and measures of the specific 
elements that make up the planning activity. 


Rockart, John F. 
The Changing Role of the Information Systems Executive: A 
Critical Success Factors Perspective 


Sloan Management Review, vol.24, no.1, pp.3-14. 
Fall 1982. 


Three of the four critical success factors (communication, 
I/S human resources, and repositioning) directly reflect 
this evolving staff-oriented role definition. Operating 
responsibility for local I/S units is increasingly being 
transferred to local line managements. The top I/S 
executive today is a "thinker, planner, and coordinator" 
rather than a direct "implementor and doer." 


Ruby, Daniel and Richard L. Zeller 
Factors affecting the Success and Failure of an Information 
System for Product Quality 
Interfaces, Vol. 8. 10.3, mobo Uaeaee 
February 1978. 


As a research strategy, the comparative case _ study 
reported here strikes a balance between large sample 
Survey techniques and one-shot case studies. It neither 
possesses the severe limitations of these forms of 
research nor makes full use of their advantages. 

Our case indicates the importance of several factors in 
MIS implementation. 


lig At the individual level, certain attitudes were 
found to be more important than others. 

on The organizational factors of complexity, formality, 
and centralization also affect implementation. 

3 Lack of involvement by system developers is not 


sufficient to ensure failure at thesvitalseune- on 
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of explaining the system to ultimate users assumed 
by some other knowledgeable person in the group. 

4. Strong management support is instrumental to system 
adoption. 


Ruby, Daniel. 
User Attitudes and Management Information System Use 


Academy of management Journal, vol.22, no.3, pp.527-538. 
September 1979. 


Overall, these results Support the established notion that 
user attitudes (or perceptions) are significant correlates 
of system use. Attitudes are less powerful in predicting 
subjective assessments of perceived worth, although the 
relationships are significant. The finding that attitudes 
are more strongly related to actual use than they are to 
measures of perceived worth has important implications. 


User concerns are critical to success of MIS, particularly 
concerns about the impact of MIS on individual 
performance. Obviously, these concerns should be 
addressed during implementation, but in some cases it may 
be too late by then. 


A system that does not help people perform their jobs is 
not likely to be received favorably in spite of careful 
implementation efforts. A system that reduces rewards for 


users is likely to meet with disaster. A logical and 
often recommended approach to systems design is to involve 
users in the design effort. Systems designers and 


implementation teams would be well advised to find some 
means of addressing these concerns during MIS development. 


Snitkin, Sidney R. and William R. King 
Determinants of the Effectiveness of Personal Decision 
Support Systems 


Information and Management, vol.10, no.2, pp.83-89. 
January 1986. 


The overall effectiveness of the system was assessed as 
1s typical in such studies, using a perceptual measure. 
Overall, this study presents an interesting perspective 
on the evolving notion of a "personal DSS." The evidence 
that is analyzed here suggest that the pragmatic notion 
of personal DSSs has meaning and that some determinants 
of the effectiveness of such systems can be identified. 
While such systems do not meet the formal criteria for 
DSSs as they were initially conceived, they are important 
to the everyday functioning of business organizations, and 
as such. 


Var 


Srinivasan, Ananth,. 
Alternative Measures of System Effectiveness: Associations 
and Implications 


Mis Quarterly, vol gyno .> pp .245-205- 
September 1985. 


While acknowledging the importance of economic analysis 
of MIS value, researchers responds to the shifting 
emphasis from efficiency to user effectiveness by focusing 
either on MIS usage or user perceived effectiveness. 


From a practitioners perspective, it is important to 
understand what exactly is being measured when a system 
effectiveness study is initiated in an organization. 


Sullivan, Robert S. and Stephen C. Secrest 
A Simple Optimization DSS for Productivity Planning at 
Dairyman’s Cooperative Creamery Association 
Interfaces, VOdeol 5, oO some aor oS 
September-October 1985. 


The milk flow analysis program, used for daily production 
planning and inventory forecasting, relieves plant 
supervisors of about four hours a day of overwhelming hand 
calculations. 


In addition, the system has improved interaction between 
all levels of management. 


The managers of milk processing plants in the United 
States rarely use computer-based production planning and 
decision support system. 


The benefits from using MFAP are obvious but somewhat 
difficult to quantify (Then they aren’t obvious). These 
benefits are the result of improved scheduling on the 
evaporator, the bottle-neck operation. MFAP has increased 
daily plant throughput by an estimated 150,000 pounds per 
day. 


SWansOn- ee SbUBEON se 
Management Information Systems: Appreciation and 
Involvement. 


Management Sciences, vol.21, no.2, pp.178-188. 
October 1974. 


It seems reasonable to conclude that MIS appreciation 
coproduces (and is coproduced by) MIS involvement. 
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Table F-1: Research Design Types 


APPENDIX F 


TABLES OF FINDINGS 


1 2 3 Campbell 
Pre Comp Random & Stanley 
Study Year Test Grp Assign Design Type 
i Colton 1973 One shot case study 
2 Lucas 1973 X Static group comparison 
3 Swanson 1974 One shot case study 
4 Gallagher 1974 One shot case study 
5 Lucas 1975 One shot case study 
6 Lucas 1975 One shot case study 
7 Hall & Lincoln 1976 X Static group comparison 
8 Barkin & Dickson 1977 X Static group comparison 
9 Edstrom 1977 One shot case study 
10 King & Rodriguez 1978 X X X True experiment 
ll Robey & Zeller 1978 X Static group comparison 
12 Mason 1978 One shot case study 
13 Lucas 1978 X X X True experiment 
14 Matlin 1979 One shot case study 
15 Neumann & Segev 1979 One shot case study 
16 Robey 1979 One shot case study 
17 Mansour & Watson 1980 One shot case study 
18 Ball 1980 X One group pretest/posttest 
19 Larcker & Lessig 1980 One shot case study 
20 Edelman 1981 X Static group comparison 
21 Altmeyer & Bozeman 1981 One shot case study 
22 Rockart 1982 One shot case study 
23 Epstein & King 1982 X Static group comparison 
24 Feurst & Cheney 1982 One shot case study 
Zr Foppel 1982 One shot case study 
26 Bailey & Pearson 1983 One shot case study 
27 Cron & Sobol 1983 One shot case study 
28 King & Epstein 1983 One shot case study 
29 Bruwer 1984 One shot case study 
30 Decker & Plumlee 1985 One shot case study 
31 Englander & Englander 1985 X Time series 
32 Srinivasan 1985 One shot case study 
33 Sullivan & Secrest 1985 X One group pretest/posttest 
34 Hogue & Watson Eke, One shot case study 
ao loncedn 1986 X Static group comparison 
36 Franz & Robey 1986 One shot case study 
37 Kapsales 1986 One shot case study 
38 Snitkins & King 1986 One shot case study 
39 Aldag & Power 1986 X X Postrest only with control qroup 
40 Miller & Doyle 1987 One shot case study 
41 Durand et al 1987 X X Posttest only with control group 
42 Raghunathan 1988 One shot case study 
43 Baroudi & Orlikowski 1988 One shot case study 
44 Money et al 1988 One shot case study 
45 Gallupe et al 1988 X Static group comparison 


Lay 


Table F-2: Threats to Internal Validity 


Sel 
STUDY YEAR His Mat Tst Ins Reg Sel Mor Mat 

1 ‘Colton 1973 amie -1 -1 

2 Lucas 1974 +1 to +1. ot].  -1 eee 

3 Swanson 1974 ad oe a 

4 Gallagher 1974 = 2! eal = 

5 Lucas 1975 -l =1 = 2h aol 

6 Lucas 1975 ie 1 =i aul 

7 Hall & Lincoln 1976 +1 2? +1 #¢1 #41 =+:-1 =«#-1 ca 

8 Barkin & Dickson 1977 +1 2 +1 +1 =o) 

9 Edstrom 1977 ae 4k =a al 
10 King & Rodriguez 1978 +1 +1 #¢+1 #421 «¢:-421 «#4+21¢—~«421 +1 
ll Robey & Zeller 1978 +1 ? +1 41 «4-1 cl 
12 Mason 1978 co) ee! =i ae 
13 Lucas 1978 +1 +1 +1 +1 +1 +1 +1 
14 Matlin 1979 ee oe ane 
15 Neumann & Segev 1979 = =k al 
16 Robey 1979 Sl eat all =. 
17 Mansour & Watson 1980 StL il al. =k 
18 Ball 1980 ee ? +1 #«+21 a 
19 Larcker & Lessig 1980 =1, “ek = =a 
20 Edelman 1981 +1 ? +1 #++¢1 «+41 4-1 «#1 -1 
21 Altmeyer & Bozeman 1981 -1 -1 =u =e 
22 Rockart 1982 all =. = 
23 Epstein & King 1982 ra ? +1 #41 #41 ~¢-1 «1 = 
24 Feurst & Cheney 1982 = ed all =a 
25 Poppel 1982 aie et a = 
26 Bailey & Pearson 1983 a ae alt = 
27 Cron & Sobol 1983 ab al = dl 
28 King & Epstein 1983 a eg =A = 
29 Bruwer 1984 ee al —_ 
30 Decker & Plumlee 1985 a il euil =i. 
31 Englander & Englander 1985 ieee eaten e +1 +180 +1 
32 Srinivasan 1985 =| 1: aks 
33 Sullivan & Secrest 1985 = re [etl ete =a 
34 Hogue & Watson 1985 = ae =a 
S5 gba ncoeiln 1986 +1 ? +1 #41 #¢4+21 ~;-1 «1 a 
36 Franz & Robey 1986 =1- =i (| =F 
37 Kapsales 1986 = =i. a) 
38 Snitkin & King 1986 coe) a a i 
39 Aldag & Power 1986 +1 EL ee +1 
40 Miller & Doyle 1987 eo -1 -1 
41 Durand et al 1987 +1 +1 #$¢1 #t1 +1 «+;4+1 «421 +1 
42 Raghunathan 1988 =e sad =a 
43 Baroudi & Orlikowski 1988 cade tal al cdl 
44 Money et al 1988 =) ed =e ee! 
45 Gallupe et al 1988 +1 ? +1 #+¢1 #++1 ~+-1 «-1 ail. 
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Table F-3: Threats to External Validity 


Interactions: Multiple 
Test Selct Reactv xX 
Study Year & X & X Arrg Interf 
1 Colton 1973 =a 
2 Lucas 1974 =) 
3 Swanson 1974 -1 
4 Gallagher 1974 =, 
5 Lucas 1975 =] 
6 Lucas 1975 = al 
7 Hall & Lincoln 1976 = 
8 Barkin & Dickson 1977 Tn 
9 Edstrom 1977 =-1 
10 King & Rodriguez 1978  =1 ? ? 
11 Robey & Zeller 1978 =-1 
12 Mason 1978 =i 
13 Lucas 1978 -1 ? ? 
14 Matlin 1979 -1 
15 Neumann & Segev 1979 al 
16 Robey 1979 = 
17? Mansour & Watson 1980 =i! 
18 Ball 1980 -1 -1 ? 
19 Larcker & Lessig 1980 =i 
20 Edelman 1981 -1 
21 Altmeyer & Bozeman 1981 =1 
22 Rockart 1982 all 
23 Epstein & King 1982 =i 
24 Feurst & Cheney 1982 =a 
25 Poppel 1982 =1 
26 Bailey & Pearson 1983 il 
27 Cron & Sobol 1983 -1 
28 King & Epstein 1983 ial 
29 Bruwer 1984 aol 
30 Decker & Plumlee 1985 -1 
31 Englander & Englander 1985 -1 ? ? 
32 Srinivasan 1985 =] 
33 Sullivan & Secrest 159 8.5 = 1 al ? 
34 Hogue & Watson 1985 = 
35 Lincoln 1986 -1 
36 Franz & Robey 1986 alt 
37 Kapsales 1986 =i 
38 Snitkin & King 1986 =i 
39 Aldag & Power 1986 +1 ? ? 
40 Miller & Doyle 1987 =] 
41 Durand et al 1987) +1 ? ? 
42 Raghunathan 1988 -1 
43 Baroudi & Orlikowski 1988 a 
44 Money et al 1988 =I 
45 Gallupe et al 1988 =I 
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Table F-4: Handling of Time Dimension 


STUDY 


wo owtn us WD eH 


Colton 

Lucas 

Swanson 
Gallagher 

Lucas 

Lucas 

Hall & Lincoln 
Barkin & Dickson 
Edstrom 

King & Rodriguez 
Robey & Zeller 
Mason 

Lucas 

Matlin 

Neumann & Segev 
Robey 

Mansour & Watson 
Ball 

Larcker & Lessig 
Edelman 
Altmeyer & Bozeman 
Rockart 

Epstein & King 
Feurst & Cheney 
Poppel 

Bailey & Pearson 
Cron & Sobol 
King & Epstein 
Bruwer 

Decker & Plumlee 


Englander & Englander 


Srinivasan 
Sullivan & Secrest 
Hogue & Watson 
Lincoln 

Franz & Robey 
Kapsales 

Snitkin & King 
Aldag & Power 
Miller & Doyle 
Durand et al 
Raghunathan 
Baroudi & Orlikowski 
Money et al 
Gallupe et al 


YEAR 


1973 
1974 
1974 
1974 
1975 
WEES, 
1976 
1977 
197% 
1978 
1978 
1978 
1978 
1979 
1979 
1979 
1980 
1980 
1980 
1981 
1981 
1982 
1982 
1982 
1982 
1983 
1983 
1983 
1984 
1985 
Shs, 
1985 
1985 
1985 
1986 
1986 
1986 
1986 
1986 
1987 
1987 
1988 
1988 
1988 
1988 
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Handled 
Time 
Dimension: 


No 
No 
No 
No 
No 
No 
No 
No 
No 
Yes 
No 
No 
Yes 
No 
No 
No 
No 
Yes 
No 
No 
No 
No 
No 
No 
No 
No 
No 
No 
No 
No 
Yes 
No 
Yes 
No 
No 
No 
No 
No 
No 
No 
No 
No 
No 
No 
No 


Table F-5: Data Collection Techniques 


lst 2nd 
Method Method 
Study Year Used Used 
1 Colton 1973 Mailed Questionnaire Interview Survey 
2 Lucas 1974 Questionnaire 
3 Swanson 1974 Archival Questionnaire 
4 Gallagher 1974 Questionnaire Smantic Differential 
5 Lucas 1975 Questionnaire 
6 Lucas 1975 Mailed Questionnaire Archival 
7 Hall & Lincoln 1976 Interview Survey 
8 Barkin & Dickson 1977 Questionnaire Simulation 
9 Edstrom 1977 Interview Survey 
10 King & Rodriguez 1978 Questionnaire 
ll Robey & Zeller 1978 Questionnaire 
12 Mason 1978 Questionnaire? 
13 Lucas 1978 Mailed Questionnaire Questionnaire 
14 Matlin 1979 Archival 
15 Neumann & Segev 1979 Questionnaire 
16 Robey 1979 Archival Mailed Questionnaire 
17 Mansour & Watson 1980 Mailed Questionnaire 
18 Ball 1980 Questionnaire Archival 
19 Larcker & Lessig 1980 Questionnaire 
20 Edelman 1981 Archival 
21 Altmeyer & Bozeman 1981 Phone Interview Mailed Questionnaire 
22 Rockart 1982 Interview Survey 
23 Epstein & King 1982 Questionnaire 
24 Feurst & Cheney 1982 Mailed Questionnaire Mailed Questionnaire 
25 Poppel 1982 Diary Questionnaire 
26 Bailey & Pearson 1983 Interview Survey Mailed Questionnaire 
27 Cron & Sobol 1983 Questionnaire 
28 King & Epstein 1983 Questionnaire 
29 Bruwer 1984 Questionnaire 
30 Decker & Plumlee 1985 Archival 
31 Englander & Englander 1985 Archival 
32 Srinivasan 1985 Mailed Questionnaire 
33 Sullivan & Secrest 1985 Archival 
34 Hogue & Watson 1985 Interview Survey Questionnaire 
32. Lincoln 1986 Questionnaire 
36 Franz & Robey 1986 Questionnaire 
37 Kapsales 1986 Archival 
38 Snitkin & King 1986 Mailed Questionnaire 
39 Aldag & Power 1986 Laboratory Experiment Questionnaire 
40 Miller & Doyle 1987 Mailed Questionnaire 
41 Durand et al 1987 Laboratory Experiment 
42 Raghunathan 1988 Mailed Questionnaire 
43 Baroudi & Orlikowski 1988 Mailed Questionnaire 
44 Money et al 1988 Questionnaire 
45 Gallupe et al 1988 Panel Questionnaire 


sa 


Table F-6: Definitions of Effectiveness 


Study Year Definition of System Payoff 
1 Colton 1973 System usage 
2 Lucas 1974 Attitude of users toward system 
3 Swanson 1974 System usage 
4 Gallagher 1974 Dollar value information 
5 Lucas 1975 System usage 
6 Lucas 1975 System usage 
7 Hall & Lincoln 1976 Cost savings 
8 Barkin & Dickson 1977 Information usage 
9 Edstrom 1977 User influence 
10 King & Rodriguez 1978 Attitude of users toward system 
ll Robey & Zeller 1978 Attitude of users toward system 
12 Mason 1978 Problem solving ability 
13 Lucas 1978 System usage 
14 Matlin 1979 Value/cost ratio 
15 Neumann & Segev 1979 Attitude of users toward system 
16 Robey 1979 System value 
17 Mansour & Watson 1980 Performance factors 
18 Ball 1980 Attitude of users toward system 
19 Larcker & Lessig 1980 Information usefulness 
20 Edelman 1981 Cost avoidance 
21 Altmeyer & Bozeman 1981 Attitude of users toward system 
22 Rockart 1982 Job factors 
23 Epstein & King 1982 Attitude of users toward system 
24 Feurst & Cheney 1982 System usage 
25 Poppel 1982 Office automation 
26 Bailey & Pearson 1983 Attitude of users toward system 
27 Cron & Sobol 1983 Return on investment 
28 King & Epstein 1983 Information value 
29 Bruwer 1984 Attitude of users toward system 
30 Decker & Plumlee 1985 Not clear 
31 Englander & Englander 1985 Cost savings 
32 Srinivasan 1985 Attitude of users toward system 
33 Sullivan & Secrest 1985 Profit 
34 Hogue & Watson 1985 System usage 
35 Linécoln 1986 Cost benefit 
36 Franz & Robey 1986 System usefulness 
37 Kapsales 1986 Cost avoidance 
38 Snitkin & King 1986 Attitude of users toward system 
39 Aldag & Power 1986 Decision quality 
40 Miller & Doyle 1987 Attitude of users toward system 
41 Durand et al 1987 Grammar 
42 Raghunathan 1988 Implementation of plan 
43 Baroudi & Orlikowski 1988 Attitude of users toward system 
44 Money et al 1988 Attitude of users toward system 
45 Gallupe et al 1988 Decision quality 


rs 2 


Reported 
Study Year Change 
PeColton 1973 Not reported 
2 Lucas 1974 Not reported 
3 Swanson 1974 Not reported 
4 Gallagher 1974 Not reported 
5 Lucas 1975 Not reported 
6 Lucas 1975 Not reported 
7 Hall & Lincoln 1976 25% 
8 Barkin & Dickson 1977 Not reported 
9 Edstrom a alla Not reported 
10 King & Rodriguez 1978 Not reported 
11 Robey & Zeller 1978 Not reported 
12 Mason 1978 ? 
13 Lucas 1978 Not reported 
14 Matlin 1979 eet Ss 
15 Neumann & Segev 1979 Not reported 
16 Robey 1979 Not reported 
17 Mansour & Watson 1980 Not reported 
18 Ball 1980 Not reported 
19 Larcker & Lessig 1980 Not reported 
20 Edelman 1981 1> tewZz0s 
21 Altmeyer & Bozeman 1981 Not reported 
22 Rockart 1982 Not reported 
23 Epstein & King 1982 Not reported 
24 Feurst & Cheney 1982 Not reported 
25 Poppel 1982 Not reported 
26 Bailey & Pearson 1983 Not reported 
27 Cron & Sobol 1983 ? 
28 King & Epstein 1983 Not reported 
29 Bruwer 1984 Not reported 
30 Decker & Plumlee 1985 2 
31 Englander & Englander 1985 49% 
32 Srinivasan 1985 Not reported 
33 Sullivan & Secrest 1985 $100,000 
34 Hogue & Watson 1985 Not reported 
85 Linceln 1986 13-400% 
36 Franz & Robey 1986 Not reported 
37 Kapsales 1986 102% 
38 Snitkin & King 1986 Not reported 
39 Aldag & Power 1986 No improvement following implementation 
40 Miller & Doyle 1987 Not reported 
41 Durand et al 1OoB7 Not reported 
42 Raghunathan 1988 Not reported 
43 Baroudi & Orlikowski 1988 Not applicable 
44 Money et al 1988 Not reported 
45 Gallupe et al 1988 


Table F-7: Reports of Change in System Effectiveness 


Group decision support system supported better 


quality of decisions 
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